2023-07-12T15:06:01.640 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-12T15:06:01.641 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335307 --owner scheduled_gabrioux@teuthology --description orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} --name gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi --verbose -- /home/teuthworker/archive/gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335307/orig.config.yaml 2023-07-12T15:06:01.674 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-12T15:06:01.864 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335307 branch: wip-guits-testing-4-2023-07-12-0746 description: orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} email: gabrioux@redhat.com first_in_suite: false job_id: '7335307' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-guits-testing-4-2023-07-12-0746 ceph: conf: client: debug ms: 1 global: mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: b1bd054042b7797ba5b70d145a1da31693dda494 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 cephadm: cephadm_mode: root install: ceph: flavor: default sha1: b1bd054042b7797ba5b70d145a1da31693dda494 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-testing-4-2023-07-12-0746 sha1: b1bd054042b7797ba5b70d145a1da31693dda494 owner: scheduled_gabrioux@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a sha1: b1bd054042b7797ba5b70d145a1da31693dda494 sleep_before_teardown: 0 suite: orch:cephadm suite_branch: wip-guits-testing-4-2023-07-12-0746 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_b1bd054042b7797ba5b70d145a1da31693dda494/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: b1bd054042b7797ba5b70d145a1da31693dda494 targets: smithi069.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi145.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b53f236c1077f1d295baafb9e0ca9a4b447019d7 timestamp: 2023-07-12_13:54:22 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-12T15:06:01.864 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_b1bd054042b7797ba5b70d145a1da31693dda494/qa; will attempt to use it 2023-07-12T15:06:01.865 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_b1bd054042b7797ba5b70d145a1da31693dda494/qa/tasks 2023-07-12T15:06:01.866 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-12T15:06:01.869 INFO:teuthology.task.internal:Checking packages... 2023-07-12T15:06:01.902 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'b1bd054042b7797ba5b70d145a1da31693dda494' 2023-07-12T15:06:01.902 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-12T15:06:01.903 INFO:teuthology.packaging:ref: None 2023-07-12T15:06:01.903 INFO:teuthology.packaging:tag: None 2023-07-12T15:06:01.903 INFO:teuthology.packaging:branch: wip-guits-testing-4-2023-07-12-0746 2023-07-12T15:06:01.903 INFO:teuthology.packaging:sha1: b1bd054042b7797ba5b70d145a1da31693dda494 2023-07-12T15:06:01.904 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-guits-testing-4-2023-07-12-0746 2023-07-12T15:06:02.048 INFO:teuthology.task.internal:Found packages for ceph version 18.1.2-55.gb1bd0540 2023-07-12T15:06:02.049 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-12T15:06:02.056 INFO:teuthology.task.internal:no buildpackages task found 2023-07-12T15:06:02.056 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-12T15:06:02.112 INFO:teuthology.task.internal:Saving configuration 2023-07-12T15:06:02.127 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-12T15:06:02.144 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-12T15:06:02.172 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335307', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 15:00:50.299139', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-12T15:06:02.194 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335307', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 15:00:50.300999', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-12T15:06:02.195 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-12T15:06:02.202 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2023-07-12T15:06:02.202 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2023-07-12T15:06:02.203 INFO:teuthology.run_tasks:Running task console_log... 2023-07-12T15:06:02.298 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd24bc8d160>, signals=[15]) 2023-07-12T15:06:02.299 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-12T15:06:02.305 INFO:teuthology.task.internal:Opening connections... 2023-07-12T15:06:02.305 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2023-07-12T15:06:02.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-12T15:06:02.377 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2023-07-12T15:06:02.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-12T15:06:02.448 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-12T15:06:02.463 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2023-07-12T15:06:02.485 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2023-07-12T15:06:02.485 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2023-07-12T15:06:02.544 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="Red Hat Enterprise Linux" 2023-07-12T15:06:02.545 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="8.6 (Ootpa)" 2023-07-12T15:06:02.545 INFO:teuthology.orchestra.run.smithi069.stdout:ID="rhel" 2023-07-12T15:06:02.545 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="fedora" 2023-07-12T15:06:02.545 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="8.6" 2023-07-12T15:06:02.545 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el8" 2023-07-12T15:06:02.546 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-12T15:06:02.546 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2023-07-12T15:06:02.546 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-12T15:06:02.546 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://www.redhat.com/" 2023-07-12T15:06:02.546 INFO:teuthology.orchestra.run.smithi069.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-12T15:06:02.547 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-12T15:06:02.547 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:06:02.547 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-12T15:06:02.548 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-12T15:06:02.548 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-12T15:06:02.548 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-12T15:06:02.549 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2023-07-12T15:06:02.586 DEBUG:teuthology.orchestra.run.smithi145:> uname -m 2023-07-12T15:06:02.607 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2023-07-12T15:06:02.607 DEBUG:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2023-07-12T15:06:02.666 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="Red Hat Enterprise Linux" 2023-07-12T15:06:02.666 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="8.6 (Ootpa)" 2023-07-12T15:06:02.666 INFO:teuthology.orchestra.run.smithi145.stdout:ID="rhel" 2023-07-12T15:06:02.667 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="fedora" 2023-07-12T15:06:02.667 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="8.6" 2023-07-12T15:06:02.667 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el8" 2023-07-12T15:06:02.667 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-12T15:06:02.667 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2023-07-12T15:06:02.667 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-12T15:06:02.668 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.redhat.com/" 2023-07-12T15:06:02.668 INFO:teuthology.orchestra.run.smithi145.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-12T15:06:02.668 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-12T15:06:02.668 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:06:02.668 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-12T15:06:02.668 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-12T15:06:02.668 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-12T15:06:02.669 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-12T15:06:02.669 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2023-07-12T15:06:02.694 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-12T15:06:02.705 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-12T15:06:02.738 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-12T15:06:02.739 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2023-07-12T15:06:02.742 DEBUG:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2023-07-12T15:06:02.761 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-12T15:06:02.769 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-12T15:06:02.770 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2023-07-12T15:06:02.798 DEBUG:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2023-07-12T15:06:02.835 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-12T15:06:02.917 INFO:teuthology.run_tasks:Running task kernel... 2023-07-12T15:06:02.937 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-07-12T15:06:02.938 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-07-12T15:06:02.939 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-12T15:06:02.940 DEBUG:teuthology.orchestra.run.smithi145:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-12T15:06:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-12T15:06:02.959 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2023-07-12T15:06:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-12T15:06:02.962 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2023-07-12T15:06:03.018 INFO:teuthology.orchestra.run.smithi069.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-12T15:06:03.018 INFO:teuthology.task.kernel:Running kernel on smithi069: 4.18.0-372.9.1.el8.x86_64 2023-07-12T15:06:03.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum install -y kernel 2023-07-12T15:06:03.019 INFO:teuthology.orchestra.run.smithi145.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-12T15:06:03.020 INFO:teuthology.task.kernel:Running kernel on smithi145: 4.18.0-372.9.1.el8.x86_64 2023-07-12T15:06:03.020 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum install -y kernel 2023-07-12T15:06:04.245 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:06:04.245 INFO:teuthology.orchestra.run.smithi069.stdout:Unable to read consumer identity 2023-07-12T15:06:04.310 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:06:04.310 INFO:teuthology.orchestra.run.smithi145.stdout:Unable to read consumer identity 2023-07-12T15:06:07.896 INFO:teuthology.orchestra.run.smithi069.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-07-12T15:06:08.043 INFO:teuthology.orchestra.run.smithi145.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-07-12T15:06:10.746 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 5.5 MB/s | 16 MB 00:02 2023-07-12T15:06:11.059 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 5.2 MB/s | 16 MB 00:02 2023-07-12T15:06:14.187 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 444 kB/s | 24 kB 00:00 2023-07-12T15:06:14.517 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 417 kB/s | 24 kB 00:00 2023-07-12T15:06:14.832 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-12T15:06:14.887 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:06:14.889 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-07-12T15:06:14.889 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:06:15.033 DEBUG:teuthology.orchestra.run.smithi069:> echo no | sudo yum reinstall kernel || true 2023-07-12T15:06:15.203 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-12T15:06:15.262 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:06:15.263 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2023-07-12T15:06:15.263 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:06:15.364 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:06:15.365 INFO:teuthology.orchestra.run.smithi069.stdout:Unable to read consumer identity 2023-07-12T15:06:15.406 DEBUG:teuthology.orchestra.run.smithi145:> echo no | sudo yum reinstall kernel || true 2023-07-12T15:06:15.651 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:01 ago on Wed 12 Jul 2023 03:06:14 PM UTC. 2023-07-12T15:06:15.694 INFO:teuthology.orchestra.run.smithi069.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-12T15:06:15.726 INFO:teuthology.orchestra.run.smithi069.stderr:Error: No packages marked for reinstall. 2023-07-12T15:06:15.736 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:06:15.737 INFO:teuthology.orchestra.run.smithi145.stdout:Unable to read consumer identity 2023-07-12T15:06:15.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum reinstall -y kernel || true 2023-07-12T15:06:16.002 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:02 ago on Wed 12 Jul 2023 03:06:14 PM UTC. 2023-07-12T15:06:16.042 INFO:teuthology.orchestra.run.smithi145.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-12T15:06:16.076 INFO:teuthology.orchestra.run.smithi145.stderr:Error: No packages marked for reinstall. 2023-07-12T15:06:16.099 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:06:16.099 INFO:teuthology.orchestra.run.smithi069.stdout:Unable to read consumer identity 2023-07-12T15:06:16.122 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum reinstall -y kernel || true 2023-07-12T15:06:16.362 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:02 ago on Wed 12 Jul 2023 03:06:14 PM UTC. 2023-07-12T15:06:16.401 INFO:teuthology.orchestra.run.smithi069.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-12T15:06:16.421 INFO:teuthology.orchestra.run.smithi069.stderr:Error: No packages marked for reinstall. 2023-07-12T15:06:16.451 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:06:16.451 INFO:teuthology.orchestra.run.smithi145.stdout:Unable to read consumer identity 2023-07-12T15:06:16.466 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2023-07-12T15:06:16.546 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-12T15:06:16.547 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-12T15:06:16.547 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-12T15:06:16.547 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-07-12T15:06:16.548 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-12T15:06:16.584 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2023-07-12T15:06:16.601 DEBUG:teuthology.parallel:result is None 2023-07-12T15:06:16.713 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:02 ago on Wed 12 Jul 2023 03:06:14 PM UTC. 2023-07-12T15:06:16.752 INFO:teuthology.orchestra.run.smithi145.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-12T15:06:16.774 INFO:teuthology.orchestra.run.smithi145.stderr:Error: No packages marked for reinstall. 2023-07-12T15:06:16.820 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2023-07-12T15:06:16.900 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-12T15:06:16.900 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-12T15:06:16.901 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-12T15:06:16.901 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-07-12T15:06:16.901 DEBUG:teuthology.orchestra.run.smithi145:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-12T15:06:16.934 INFO:teuthology.orchestra.run.smithi145.stdout:ttyS1 2023-07-12T15:06:16.949 DEBUG:teuthology.parallel:result is None 2023-07-12T15:06:16.949 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-12T15:06:16.956 INFO:teuthology.task.internal:Creating test directory... 2023-07-12T15:06:16.957 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-12T15:06:16.959 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-12T15:06:17.006 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-12T15:06:17.014 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-12T15:06:17.044 INFO:teuthology.task.internal:Creating archive directory... 2023-07-12T15:06:17.045 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-12T15:06:17.048 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-12T15:06:17.075 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-12T15:06:17.082 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-12T15:06:17.083 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-12T15:06:17.112 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-12T15:06:17.145 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-12T15:06:17.155 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-12T15:06:17.160 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-12T15:06:17.170 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-12T15:06:17.171 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-12T15:06:17.178 INFO:teuthology.task.internal:Configuring sudo... 2023-07-12T15:06:17.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-12T15:06:17.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-12T15:06:17.223 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-12T15:06:17.232 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-12T15:06:17.232 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-12T15:06:17.247 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-12T15:06:17.282 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-12T15:06:17.349 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-12T15:06:17.403 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-12T15:06:17.485 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-12T15:06:17.526 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:06:17.527 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-12T15:06:17.593 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-12T15:06:17.639 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-12T15:06:17.692 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-12T15:06:17.781 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-12T15:06:17.818 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-07-12T15:06:17.818 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-12T15:06:17.886 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2023-07-12T15:06:17.889 DEBUG:teuthology.orchestra.run.smithi145:> sudo service rsyslog restart 2023-07-12T15:06:17.942 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-12T15:06:17.978 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-12T15:06:18.360 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-12T15:06:18.373 INFO:teuthology.task.internal:Starting timer... 2023-07-12T15:06:18.374 INFO:teuthology.run_tasks:Running task pcp... 2023-07-12T15:06:18.446 INFO:teuthology.run_tasks:Running task selinux... 2023-07-12T15:06:18.470 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-07-12T15:06:18.471 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2023-07-12T15:06:18.530 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2023-07-12T15:06:18.530 DEBUG:teuthology.orchestra.run.smithi145:> sudo service auditd rotate 2023-07-12T15:06:18.586 INFO:teuthology.orchestra.run.smithi145.stdout:Rotating logs: 2023-07-12T15:06:18.588 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-12T15:06:18.588 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2023-07-12T15:06:18.616 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2023-07-12T15:06:18.616 DEBUG:teuthology.orchestra.run.smithi145:> /usr/sbin/getenforce 2023-07-12T15:06:18.637 INFO:teuthology.orchestra.run.smithi145.stdout:Permissive 2023-07-12T15:06:18.637 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi069.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi145.front.sepia.ceph.com': 'permissive'} 2023-07-12T15:06:18.638 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-12T15:06:18.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-12T15:06:18.686 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-12T15:06:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-12T15:06:18.715 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-12T15:06:18.715 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2023-07-12T15:06:18.754 DEBUG:teuthology.orchestra.run.smithi145:> sudo /usr/sbin/setenforce permissive 2023-07-12T15:06:18.782 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-12T15:06:18.791 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-12T15:06:18.797 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-12T15:06:18.935 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-12T15:06:18.962 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-12T15:06:18.964 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi069.front.sepia.ceph.com,smithi145.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-12T15:14:55.047 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi069.front.sepia.ceph.com'), Remote(name='ubuntu@smithi145.front.sepia.ceph.com')] 2023-07-12T15:14:55.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-07-12T15:14:55.050 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-12T15:14:55.120 DEBUG:teuthology.orchestra.run.smithi069:> true 2023-07-12T15:14:55.185 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-07-12T15:14:55.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-07-12T15:14:55.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-12T15:14:55.252 DEBUG:teuthology.orchestra.run.smithi145:> true 2023-07-12T15:14:55.321 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-07-12T15:14:55.321 INFO:teuthology.run_tasks:Running task clock... 2023-07-12T15:14:55.332 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-12T15:14:55.333 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-12T15:14:55.334 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-12T15:14:55.336 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-12T15:14:55.337 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-12T15:14:55.366 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-12T15:14:55.381 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-12T15:14:55.392 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-12T15:14:55.406 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-12T15:14:55.459 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2023-07-12T15:14:55.466 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: ntpd: command not found 2023-07-12T15:14:55.470 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2023-07-12T15:14:55.477 INFO:teuthology.orchestra.run.smithi145.stdout:506 Cannot talk to daemon 2023-07-12T15:14:55.484 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-12T15:14:55.492 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-12T15:14:55.497 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-12T15:14:55.506 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-12T15:14:55.557 INFO:teuthology.orchestra.run.smithi069.stderr:bash: ntpq: command not found 2023-07-12T15:14:55.560 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-12T15:14:55.560 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2023-07-12T15:14:55.560 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-12T15:14:55.560 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-12T15:14:55.560 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-12T15:14:55.561 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-12T15:14:55.577 INFO:teuthology.orchestra.run.smithi145.stderr:bash: ntpq: command not found 2023-07-12T15:14:55.581 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-12T15:14:55.581 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2023-07-12T15:14:55.581 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-12T15:14:55.581 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-12T15:14:55.581 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-12T15:14:55.582 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-12T15:14:55.582 INFO:teuthology.run_tasks:Running task pexec... 2023-07-12T15:14:55.592 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-12T15:14:55.593 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi069.front.sepia.ceph.com 2023-07-12T15:14:55.593 DEBUG:teuthology.orchestra.run.smithi069:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-12T15:14:55.594 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi145.front.sepia.ceph.com 2023-07-12T15:14:55.594 DEBUG:teuthology.orchestra.run.smithi145:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-12T15:14:55.972 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:14:55.994 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:14:56.763 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:26 ago on Wed 12 Jul 2023 03:13:30 PM UTC. 2023-07-12T15:14:56.784 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:26 ago on Wed 12 Jul 2023 03:13:30 PM UTC. 2023-07-12T15:14:58.526 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:14:58.527 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:14:58.528 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2023-07-12T15:14:58.528 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:14:58.528 INFO:teuthology.orchestra.run.smithi145.stdout:Resetting modules: 2023-07-12T15:14:58.528 INFO:teuthology.orchestra.run.smithi145.stdout: container-tools 2023-07-12T15:14:58.528 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:14:58.528 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:14:58.529 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:14:58.529 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:14:58.529 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:14:58.529 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:14:58.530 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2023-07-12T15:14:58.530 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:14:58.530 INFO:teuthology.orchestra.run.smithi069.stdout:Resetting modules: 2023-07-12T15:14:58.530 INFO:teuthology.orchestra.run.smithi069.stdout: container-tools 2023-07-12T15:14:58.530 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:14:58.530 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:14:58.530 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:14:58.531 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:14:59.478 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:29 ago on Wed 12 Jul 2023 03:13:30 PM UTC. 2023-07-12T15:14:59.493 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:29 ago on Wed 12 Jul 2023 03:13:30 PM UTC. 2023-07-12T15:15:02.184 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:15:02.334 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:15:02.662 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:15:02.837 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:15:03.440 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:33 ago on Wed 12 Jul 2023 03:13:30 PM UTC. 2023-07-12T15:15:03.605 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:33 ago on Wed 12 Jul 2023 03:13:30 PM UTC. 2023-07-12T15:15:05.296 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:15:05.390 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================================ 2023-07-12T15:15:05.390 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-07-12T15:15:05.390 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================================ 2023-07-12T15:15:05.390 INFO:teuthology.orchestra.run.smithi069.stdout:Installing group/module packages: 2023-07-12T15:15:05.390 INFO:teuthology.orchestra.run.smithi069.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-12T15:15:05.391 INFO:teuthology.orchestra.run.smithi069.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-12T15:15:05.391 INFO:teuthology.orchestra.run.smithi069.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-12T15:15:05.391 INFO:teuthology.orchestra.run.smithi069.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-12T15:15:05.391 INFO:teuthology.orchestra.run.smithi069.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-12T15:15:05.391 INFO:teuthology.orchestra.run.smithi069.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-12T15:15:05.391 INFO:teuthology.orchestra.run.smithi069.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-12T15:15:05.391 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-07-12T15:15:05.392 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-12T15:15:05.392 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-12T15:15:05.392 INFO:teuthology.orchestra.run.smithi069.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-12T15:15:05.392 INFO:teuthology.orchestra.run.smithi069.stdout:Installing module profiles: 2023-07-12T15:15:05.392 INFO:teuthology.orchestra.run.smithi069.stdout: container-tools/common 2023-07-12T15:15:05.392 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:15:05.392 INFO:teuthology.orchestra.run.smithi069.stdout:Enabling module streams: 2023-07-12T15:15:05.392 INFO:teuthology.orchestra.run.smithi069.stdout: container-tools rhel8 2023-07-12T15:15:05.393 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:15:05.393 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:15:05.393 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================================ 2023-07-12T15:15:05.393 INFO:teuthology.orchestra.run.smithi069.stdout:Install 10 Packages 2023-07-12T15:15:05.393 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:15:05.394 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 18 M 2023-07-12T15:15:05.394 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 65 M 2023-07-12T15:15:05.394 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:15:05.453 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:15:05.460 INFO:teuthology.orchestra.run.smithi069.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.3 MB/s | 94 kB 00:00 2023-07-12T15:15:05.507 INFO:teuthology.orchestra.run.smithi069.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 217 kB/s | 25 kB 00:00 2023-07-12T15:15:05.528 INFO:teuthology.orchestra.run.smithi069.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 302 kB/s | 41 kB 00:00 2023-07-12T15:15:05.551 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================================ 2023-07-12T15:15:05.551 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2023-07-12T15:15:05.552 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================================ 2023-07-12T15:15:05.552 INFO:teuthology.orchestra.run.smithi145.stdout:Installing group/module packages: 2023-07-12T15:15:05.552 INFO:teuthology.orchestra.run.smithi145.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-12T15:15:05.552 INFO:teuthology.orchestra.run.smithi145.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-12T15:15:05.552 INFO:teuthology.orchestra.run.smithi145.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-12T15:15:05.552 INFO:teuthology.orchestra.run.smithi145.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-12T15:15:05.553 INFO:teuthology.orchestra.run.smithi145.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-12T15:15:05.553 INFO:teuthology.orchestra.run.smithi145.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-12T15:15:05.553 INFO:teuthology.orchestra.run.smithi145.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-12T15:15:05.553 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2023-07-12T15:15:05.553 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-12T15:15:05.553 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-12T15:15:05.553 INFO:teuthology.orchestra.run.smithi145.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-12T15:15:05.553 INFO:teuthology.orchestra.run.smithi145.stdout:Installing module profiles: 2023-07-12T15:15:05.554 INFO:teuthology.orchestra.run.smithi145.stdout: container-tools/common 2023-07-12T15:15:05.554 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:15:05.554 INFO:teuthology.orchestra.run.smithi145.stdout:Enabling module streams: 2023-07-12T15:15:05.554 INFO:teuthology.orchestra.run.smithi145.stdout: container-tools rhel8 2023-07-12T15:15:05.554 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:15:05.554 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:15:05.554 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================================ 2023-07-12T15:15:05.554 INFO:teuthology.orchestra.run.smithi145.stdout:Install 10 Packages 2023-07-12T15:15:05.555 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:15:05.555 INFO:teuthology.orchestra.run.smithi069.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 536 kB/s | 49 kB 00:00 2023-07-12T15:15:05.556 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 18 M 2023-07-12T15:15:05.556 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 65 M 2023-07-12T15:15:05.556 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:15:05.586 INFO:teuthology.orchestra.run.smithi069.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.5 MB/s | 524 kB 00:00 2023-07-12T15:15:05.612 INFO:teuthology.orchestra.run.smithi069.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-07-12T15:15:05.631 INFO:teuthology.orchestra.run.smithi145.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2023-07-12T15:15:05.637 INFO:teuthology.orchestra.run.smithi069.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 4.1 MB/s | 213 kB 00:00 2023-07-12T15:15:05.656 INFO:teuthology.orchestra.run.smithi145.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 242 kB/s | 25 kB 00:00 2023-07-12T15:15:05.682 INFO:teuthology.orchestra.run.smithi145.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 319 kB/s | 41 kB 00:00 2023-07-12T15:15:05.707 INFO:teuthology.orchestra.run.smithi145.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 654 kB/s | 49 kB 00:00 2023-07-12T15:15:05.729 INFO:teuthology.orchestra.run.smithi069.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 24 MB/s | 2.2 MB 00:00 2023-07-12T15:15:05.740 INFO:teuthology.orchestra.run.smithi145.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.1 MB/s | 524 kB 00:00 2023-07-12T15:15:05.757 INFO:teuthology.orchestra.run.smithi145.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.9 MB/s | 149 kB 00:00 2023-07-12T15:15:05.791 INFO:teuthology.orchestra.run.smithi145.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 4.1 MB/s | 213 kB 00:00 2023-07-12T15:15:05.846 INFO:teuthology.orchestra.run.smithi069.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 30 MB/s | 7.0 MB 00:00 2023-07-12T15:15:05.883 INFO:teuthology.orchestra.run.smithi145.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 24 MB/s | 2.2 MB 00:00 2023-07-12T15:15:05.971 INFO:teuthology.orchestra.run.smithi069.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 18 MB/s | 8.0 MB 00:00 2023-07-12T15:15:05.972 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:15:05.972 INFO:teuthology.orchestra.run.smithi069.stdout:Total 31 MB/s | 18 MB 00:00 2023-07-12T15:15:06.000 INFO:teuthology.orchestra.run.smithi145.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 29 MB/s | 7.0 MB 00:00 2023-07-12T15:15:06.092 INFO:teuthology.orchestra.run.smithi145.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 19 MB/s | 8.0 MB 00:00 2023-07-12T15:15:06.092 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:15:06.093 INFO:teuthology.orchestra.run.smithi145.stdout:Total 34 MB/s | 18 MB 00:00 2023-07-12T15:15:06.174 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:15:06.192 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:15:06.193 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:15:06.293 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:15:06.311 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:15:06.312 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:15:06.373 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:15:06.377 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:15:06.493 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:15:06.497 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:15:06.877 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:15:07.045 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:15:07.079 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-12T15:15:07.262 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-12T15:15:07.283 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-12T15:15:07.423 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-12T15:15:07.493 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-12T15:15:07.650 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-12T15:15:07.722 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-12T15:15:07.850 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-12T15:15:08.069 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-12T15:15:08.227 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-12T15:15:08.814 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-12T15:15:09.015 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-12T15:15:09.704 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-12T15:15:09.880 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-12T15:15:09.905 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-12T15:15:10.107 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-12T15:15:10.146 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-12T15:15:10.284 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-12T15:15:10.355 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-12T15:15:10.532 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-12T15:15:10.548 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-12T15:15:10.548 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-12T15:15:10.548 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-12T15:15:10.548 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-12T15:15:10.548 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-12T15:15:10.549 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-12T15:15:10.549 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-12T15:15:10.549 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-12T15:15:10.549 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-12T15:15:10.549 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-12T15:15:10.804 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-12T15:15:10.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-12T15:15:10.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-12T15:15:10.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-12T15:15:10.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-12T15:15:10.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-12T15:15:10.806 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-12T15:15:10.806 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-12T15:15:10.806 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-12T15:15:10.806 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-12T15:15:11.645 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-12T15:15:11.645 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:41 ago on Wed 12 Jul 2023 03:13:30 PM UTC. 2023-07-12T15:15:11.953 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-12T15:15:11.953 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:41 ago on Wed 12 Jul 2023 03:13:30 PM UTC. 2023-07-12T15:15:14.584 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:15:14.781 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:15:14.781 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:15:14.781 INFO:teuthology.orchestra.run.smithi069.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-12T15:15:14.781 INFO:teuthology.orchestra.run.smithi069.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-12T15:15:14.781 INFO:teuthology.orchestra.run.smithi069.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-12T15:15:14.782 INFO:teuthology.orchestra.run.smithi069.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-12T15:15:14.782 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-12T15:15:14.782 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-12T15:15:14.782 INFO:teuthology.orchestra.run.smithi069.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-12T15:15:14.782 INFO:teuthology.orchestra.run.smithi069.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-12T15:15:14.782 INFO:teuthology.orchestra.run.smithi069.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-12T15:15:14.782 INFO:teuthology.orchestra.run.smithi069.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-12T15:15:14.783 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:15:14.783 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:15:14.852 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:15:14.917 DEBUG:teuthology.parallel:result is None 2023-07-12T15:15:15.040 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:15:15.040 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:15:15.040 INFO:teuthology.orchestra.run.smithi145.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-12T15:15:15.041 INFO:teuthology.orchestra.run.smithi145.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-12T15:15:15.041 INFO:teuthology.orchestra.run.smithi145.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-12T15:15:15.041 INFO:teuthology.orchestra.run.smithi145.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-12T15:15:15.041 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-12T15:15:15.041 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-12T15:15:15.041 INFO:teuthology.orchestra.run.smithi145.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-12T15:15:15.041 INFO:teuthology.orchestra.run.smithi145.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-12T15:15:15.042 INFO:teuthology.orchestra.run.smithi145.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-12T15:15:15.042 INFO:teuthology.orchestra.run.smithi145.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-12T15:15:15.042 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:15:15.042 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:15:15.194 DEBUG:teuthology.parallel:result is None 2023-07-12T15:15:15.195 INFO:teuthology.run_tasks:Running task install... 2023-07-12T15:15:15.203 DEBUG:teuthology.task.install:project ceph 2023-07-12T15:15:15.203 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b1bd054042b7797ba5b70d145a1da31693dda494'}} 2023-07-12T15:15:15.204 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'b1bd054042b7797ba5b70d145a1da31693dda494'} 2023-07-12T15:15:15.204 INFO:teuthology.task.install:Using flavor: default 2023-07-12T15:15:15.209 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-12T15:15:15.210 INFO:teuthology.task.install:extra packages: [] 2023-07-12T15:15:15.210 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b1bd054042b7797ba5b70d145a1da31693dda494', 'tag': None, 'wait_for_package': False} 2023-07-12T15:15:15.211 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b1bd054042b7797ba5b70d145a1da31693dda494 2023-07-12T15:15:15.213 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b1bd054042b7797ba5b70d145a1da31693dda494', 'tag': None, 'wait_for_package': False} 2023-07-12T15:15:15.213 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b1bd054042b7797ba5b70d145a1da31693dda494 2023-07-12T15:15:15.345 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-guits-testing-4-2023-07-12-0746/b1bd054042b7797ba5b70d145a1da31693dda494/centos/8/flavors/default/ 2023-07-12T15:15:15.346 INFO:teuthology.task.install.rpm:Package version is 18.1.2-55.gb1bd0540 2023-07-12T15:15:15.361 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-guits-testing-4-2023-07-12-0746/b1bd054042b7797ba5b70d145a1da31693dda494/centos/8/flavors/default/ 2023-07-12T15:15:15.362 INFO:teuthology.task.install.rpm:Package version is 18.1.2-55.gb1bd0540 2023-07-12T15:15:15.454 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-guits-testing-4-2023-07-12-0746/b1bd054042b7797ba5b70d145a1da31693dda494/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-guits-testing-4-2023-07-12-0746/b1bd054042b7797ba5b70d145a1da31693dda494/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-guits-testing-4-2023-07-12-0746/b1bd054042b7797ba5b70d145a1da31693dda494/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-12T15:15:15.454 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:15:15.455 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-12T15:15:15.483 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-07-12T15:15:15.483 DEBUG:teuthology.orchestra.run.smithi069:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b1bd054042b7797ba5b70d145a1da31693dda494/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-12T15:15:15.526 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-guits-testing-4-2023-07-12-0746/b1bd054042b7797ba5b70d145a1da31693dda494/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-guits-testing-4-2023-07-12-0746/b1bd054042b7797ba5b70d145a1da31693dda494/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-guits-testing-4-2023-07-12-0746/b1bd054042b7797ba5b70d145a1da31693dda494/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-12T15:15:15.526 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-07-12T15:15:15.527 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-12T15:15:15.551 DEBUG:teuthology.orchestra.run.smithi069:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-12T15:15:15.554 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-07-12T15:15:15.554 DEBUG:teuthology.orchestra.run.smithi145:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b1bd054042b7797ba5b70d145a1da31693dda494/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-12T15:15:15.621 DEBUG:teuthology.orchestra.run.smithi145:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-12T15:15:15.628 DEBUG:teuthology.orchestra.run.smithi069:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-12T15:15:15.695 INFO:teuthology.orchestra.run.smithi069.stdout:check_obsoletes = 1 2023-07-12T15:15:15.696 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum clean all 2023-07-12T15:15:15.697 DEBUG:teuthology.orchestra.run.smithi145:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-12T15:15:15.763 INFO:teuthology.orchestra.run.smithi145.stdout:check_obsoletes = 1 2023-07-12T15:15:15.764 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum clean all 2023-07-12T15:15:16.066 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:15:16.137 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:15:16.403 INFO:teuthology.orchestra.run.smithi069.stdout:44 files removed 2023-07-12T15:15:16.450 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw 2023-07-12T15:15:16.452 INFO:teuthology.orchestra.run.smithi145.stdout:44 files removed 2023-07-12T15:15:16.499 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-radosgw 2023-07-12T15:15:16.780 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:15:16.829 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:15:17.570 INFO:teuthology.orchestra.run.smithi145.stdout:ceph packages for x86_64 247 kB/s | 80 kB 00:00 2023-07-12T15:15:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:ceph packages for x86_64 201 kB/s | 80 kB 00:00 2023-07-12T15:15:17.923 INFO:teuthology.orchestra.run.smithi069.stdout:ceph noarch packages 33 kB/s | 9.4 kB 00:00 2023-07-12T15:15:17.962 INFO:teuthology.orchestra.run.smithi145.stdout:ceph noarch packages 26 kB/s | 9.4 kB 00:00 2023-07-12T15:15:18.183 INFO:teuthology.orchestra.run.smithi069.stdout:ceph source packages 6.3 kB/s | 1.5 kB 00:00 2023-07-12T15:15:18.363 INFO:teuthology.orchestra.run.smithi145.stdout:ceph source packages 3.9 kB/s | 1.5 kB 00:00 2023-07-12T15:15:20.583 INFO:teuthology.orchestra.run.smithi069.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-07-12T15:15:20.955 INFO:teuthology.orchestra.run.smithi145.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2023-07-12T15:15:21.445 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2023-07-12T15:15:22.174 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-07-12T15:15:24.834 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 377 kB/s | 24 kB 00:00 2023-07-12T15:15:25.614 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 397 kB/s | 24 kB 00:00 2023-07-12T15:15:25.752 INFO:teuthology.orchestra.run.smithi069.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 MB/s | 53 MB 00:00 2023-07-12T15:15:26.474 INFO:teuthology.orchestra.run.smithi145.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 60 MB/s | 47 MB 00:00 2023-07-12T15:15:33.820 INFO:teuthology.orchestra.run.smithi069.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 61 MB/s | 47 MB 00:00 2023-07-12T15:15:34.533 INFO:teuthology.orchestra.run.smithi145.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 MB/s | 53 MB 00:00 2023-07-12T15:15:44.729 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:15:44.732 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================================== 2023-07-12T15:15:44.732 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-07-12T15:15:44.732 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================================== 2023-07-12T15:15:44.732 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:15:44.733 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 13 M 2023-07-12T15:15:44.733 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrading: 2023-07-12T15:15:44.733 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 3.7 M 2023-07-12T15:15:44.733 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1 x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 3.6 M 2023-07-12T15:15:44.733 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-07-12T15:15:44.733 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 5.4 M 2023-07-12T15:15:44.734 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 22 M 2023-07-12T15:15:44.734 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 24 k 2023-07-12T15:15:44.734 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-12T15:15:44.734 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2 x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 770 k 2023-07-12T15:15:44.734 INFO:teuthology.orchestra.run.smithi069.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-12T15:15:44.734 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-12T15:15:44.735 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1 x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 475 k 2023-07-12T15:15:44.735 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-12T15:15:44.735 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2 x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 6.5 M 2023-07-12T15:15:44.735 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-12T15:15:44.735 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-12T15:15:44.735 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 45 k 2023-07-12T15:15:44.735 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 119 k 2023-07-12T15:15:44.735 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 190 k 2023-07-12T15:15:44.736 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-12T15:15:44.736 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 371 k 2023-07-12T15:15:44.736 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 364 k 2023-07-12T15:15:44.736 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 107 k 2023-07-12T15:15:44.736 INFO:teuthology.orchestra.run.smithi069.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-12T15:15:44.736 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:15:44.736 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:15:44.737 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================================== 2023-07-12T15:15:44.737 INFO:teuthology.orchestra.run.smithi069.stdout:Install 21 Packages 2023-07-12T15:15:44.737 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrade 2 Packages 2023-07-12T15:15:44.737 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:15:44.737 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 59 M 2023-07-12T15:15:44.738 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:15:45.815 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:15:45.817 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================================== 2023-07-12T15:15:45.817 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2023-07-12T15:15:45.817 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================================== 2023-07-12T15:15:45.818 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:15:45.818 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 13 M 2023-07-12T15:15:45.818 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrading: 2023-07-12T15:15:45.818 INFO:teuthology.orchestra.run.smithi145.stdout: librados2 x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 3.7 M 2023-07-12T15:15:45.818 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1 x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 3.6 M 2023-07-12T15:15:45.818 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2023-07-12T15:15:45.818 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 5.4 M 2023-07-12T15:15:45.819 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 22 M 2023-07-12T15:15:45.819 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 24 k 2023-07-12T15:15:45.819 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-12T15:15:45.819 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2 x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 770 k 2023-07-12T15:15:45.819 INFO:teuthology.orchestra.run.smithi145.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-12T15:15:45.819 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-12T15:15:45.819 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1 x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 475 k 2023-07-12T15:15:45.820 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-12T15:15:45.820 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2 x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 6.5 M 2023-07-12T15:15:45.820 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-12T15:15:45.820 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-12T15:15:45.820 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 45 k 2023-07-12T15:15:45.820 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 119 k 2023-07-12T15:15:45.820 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 190 k 2023-07-12T15:15:45.821 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-12T15:15:45.821 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 371 k 2023-07-12T15:15:45.821 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 364 k 2023-07-12T15:15:45.821 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 107 k 2023-07-12T15:15:45.821 INFO:teuthology.orchestra.run.smithi145.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-12T15:15:45.821 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:15:45.821 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:15:45.821 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================================== 2023-07-12T15:15:45.822 INFO:teuthology.orchestra.run.smithi145.stdout:Install 21 Packages 2023-07-12T15:15:45.822 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrade 2 Packages 2023-07-12T15:15:45.822 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:15:45.822 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 59 M 2023-07-12T15:15:45.823 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:15:46.029 INFO:teuthology.orchestra.run.smithi069.stdout:(1/23): ceph-base-18.1.2-55.gb1bd0540.el8.x86_6 5.6 MB/s | 5.4 MB 00:00 2023-07-12T15:15:46.095 INFO:teuthology.orchestra.run.smithi069.stdout:(2/23): ceph-selinux-18.1.2-55.gb1bd0540.el8.x8 366 kB/s | 24 kB 00:00 2023-07-12T15:15:46.229 INFO:teuthology.orchestra.run.smithi069.stdout:(3/23): libcephfs2-18.1.2-55.gb1bd0540.el8.x86_ 5.6 MB/s | 770 kB 00:00 2023-07-12T15:15:46.321 INFO:teuthology.orchestra.run.smithi069.stdout:(4/23): libradosstriper1-18.1.2-55.gb1bd0540.el 5.1 MB/s | 475 kB 00:00 2023-07-12T15:15:47.038 INFO:teuthology.orchestra.run.smithi069.stdout:(5/23): librgw2-18.1.2-55.gb1bd0540.el8.x86_64. 9.1 MB/s | 6.5 MB 00:00 2023-07-12T15:15:47.121 INFO:teuthology.orchestra.run.smithi069.stdout:(6/23): python3-ceph-argparse-18.1.2-55.gb1bd05 536 kB/s | 45 kB 00:00 2023-07-12T15:15:47.205 INFO:teuthology.orchestra.run.smithi069.stdout:(7/23): python3-ceph-common-18.1.2-55.gb1bd0540 1.4 MB/s | 119 kB 00:00 2023-07-12T15:15:47.297 INFO:teuthology.orchestra.run.smithi069.stdout:(8/23): python3-cephfs-18.1.2-55.gb1bd0540.el8. 2.0 MB/s | 190 kB 00:00 2023-07-12T15:15:47.405 INFO:teuthology.orchestra.run.smithi069.stdout:(9/23): python3-rados-18.1.2-55.gb1bd0540.el8.x 3.3 MB/s | 371 kB 00:00 2023-07-12T15:15:47.531 INFO:teuthology.orchestra.run.smithi069.stdout:(10/23): python3-rbd-18.1.2-55.gb1bd0540.el8.x8 2.9 MB/s | 364 kB 00:00 2023-07-12T15:15:47.614 INFO:teuthology.orchestra.run.smithi069.stdout:(11/23): python3-rgw-18.1.2-55.gb1bd0540.el8.x8 1.2 MB/s | 107 kB 00:00 2023-07-12T15:15:47.731 INFO:teuthology.orchestra.run.smithi069.stdout:(12/23): gperftools-libs-2.7-9.el8.x86_64.rpm 2.6 MB/s | 306 kB 00:00 2023-07-12T15:15:47.765 INFO:teuthology.orchestra.run.smithi069.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 1.7 MB/s | 59 kB 00:00 2023-07-12T15:15:47.815 INFO:teuthology.orchestra.run.smithi069.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2023-07-12T15:15:47.965 INFO:teuthology.orchestra.run.smithi069.stdout:(15/23): ceph-radosgw-18.1.2-55.gb1bd0540.el8.x 4.5 MB/s | 13 MB 00:02 2023-07-12T15:15:48.033 INFO:teuthology.orchestra.run.smithi069.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 702 kB/s | 47 kB 00:00 2023-07-12T15:15:48.058 INFO:teuthology.orchestra.run.smithi145.stdout:(1/23): ceph-base-18.1.2-55.gb1bd0540.el8.x86_6 2.6 MB/s | 5.4 MB 00:02 2023-07-12T15:15:48.074 INFO:teuthology.orchestra.run.smithi069.stdout:(17/23): thrift-0.13.0-2.el8.x86_64.rpm 6.7 MB/s | 1.7 MB 00:00 2023-07-12T15:15:48.100 INFO:teuthology.orchestra.run.smithi069.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.8 MB/s | 259 kB 00:00 2023-07-12T15:15:48.158 INFO:teuthology.orchestra.run.smithi069.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 525 kB/s | 44 kB 00:00 2023-07-12T15:15:48.184 INFO:teuthology.orchestra.run.smithi069.stdout:(20/23): librdkafka-0.11.4-3.el8.x86_64.rpm 4.4 MB/s | 354 kB 00:00 2023-07-12T15:15:48.267 INFO:teuthology.orchestra.run.smithi145.stdout:(2/23): ceph-radosgw-18.1.2-55.gb1bd0540.el8.x8 5.6 MB/s | 13 MB 00:02 2023-07-12T15:15:48.301 INFO:teuthology.orchestra.run.smithi145.stdout:(3/23): ceph-selinux-18.1.2-55.gb1bd0540.el8.x8 100 kB/s | 24 kB 00:00 2023-07-12T15:15:48.417 INFO:teuthology.orchestra.run.smithi145.stdout:(4/23): libcephfs2-18.1.2-55.gb1bd0540.el8.x86_ 5.0 MB/s | 770 kB 00:00 2023-07-12T15:15:48.451 INFO:teuthology.orchestra.run.smithi145.stdout:(5/23): libradosstriper1-18.1.2-55.gb1bd0540.el 3.1 MB/s | 475 kB 00:00 2023-07-12T15:15:48.543 INFO:teuthology.orchestra.run.smithi145.stdout:(6/23): python3-ceph-argparse-18.1.2-55.gb1bd05 487 kB/s | 45 kB 00:00 2023-07-12T15:15:48.651 INFO:teuthology.orchestra.run.smithi145.stdout:(7/23): python3-ceph-common-18.1.2-55.gb1bd0540 1.1 MB/s | 119 kB 00:00 2023-07-12T15:15:48.727 INFO:teuthology.orchestra.run.smithi145.stdout:(8/23): python3-cephfs-18.1.2-55.gb1bd0540.el8. 2.5 MB/s | 190 kB 00:00 2023-07-12T15:15:48.809 INFO:teuthology.orchestra.run.smithi069.stdout:(21/23): librados2-18.1.2-55.gb1bd0540.el8.x86_ 5.6 MB/s | 3.7 MB 00:00 2023-07-12T15:15:48.844 INFO:teuthology.orchestra.run.smithi145.stdout:(9/23): python3-rados-18.1.2-55.gb1bd0540.el8.x 3.1 MB/s | 371 kB 00:00 2023-07-12T15:15:48.961 INFO:teuthology.orchestra.run.smithi145.stdout:(10/23): python3-rbd-18.1.2-55.gb1bd0540.el8.x8 3.0 MB/s | 364 kB 00:00 2023-07-12T15:15:49.019 INFO:teuthology.orchestra.run.smithi145.stdout:(11/23): python3-rgw-18.1.2-55.gb1bd0540.el8.x8 1.8 MB/s | 107 kB 00:00 2023-07-12T15:15:49.025 INFO:teuthology.orchestra.run.smithi069.stdout:(22/23): librbd1-18.1.2-55.gb1bd0540.el8.x86_64 4.2 MB/s | 3.6 MB 00:00 2023-07-12T15:15:49.345 INFO:teuthology.orchestra.run.smithi145.stdout:(12/23): librgw2-18.1.2-55.gb1bd0540.el8.x86_64 7.0 MB/s | 6.5 MB 00:00 2023-07-12T15:15:49.401 INFO:teuthology.orchestra.run.smithi069.stdout:(23/23): ceph-common-18.1.2-55.gb1bd0540.el8.x8 5.0 MB/s | 22 MB 00:04 2023-07-12T15:15:49.404 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:15:49.404 INFO:teuthology.orchestra.run.smithi069.stdout:Total 13 MB/s | 59 MB 00:04 2023-07-12T15:15:49.444 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:15:49.488 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:15:49.488 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:15:49.826 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:15:49.830 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:15:49.928 INFO:teuthology.orchestra.run.smithi145.stdout:(13/23): gperftools-libs-2.7-9.el8.x86_64.rpm 337 kB/s | 306 kB 00:00 2023-07-12T15:15:49.987 INFO:teuthology.orchestra.run.smithi145.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 92 kB/s | 59 kB 00:00 2023-07-12T15:15:50.037 INFO:teuthology.orchestra.run.smithi145.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 690 kB/s | 75 kB 00:00 2023-07-12T15:15:50.096 INFO:teuthology.orchestra.run.smithi145.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.3 MB/s | 259 kB 00:00 2023-07-12T15:15:50.130 INFO:teuthology.orchestra.run.smithi145.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 1.3 MB/s | 44 kB 00:00 2023-07-12T15:15:50.172 INFO:teuthology.orchestra.run.smithi145.stdout:(18/23): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-07-12T15:15:50.205 INFO:teuthology.orchestra.run.smithi145.stdout:(19/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2023-07-12T15:15:50.731 INFO:teuthology.orchestra.run.smithi145.stdout:(20/23): librados2-18.1.2-55.gb1bd0540.el8.x86_ 7.0 MB/s | 3.7 MB 00:00 2023-07-12T15:15:50.998 INFO:teuthology.orchestra.run.smithi145.stdout:(21/23): librbd1-18.1.2-55.gb1bd0540.el8.x86_64 13 MB/s | 3.6 MB 00:00 2023-07-12T15:15:51.038 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:15:51.261 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-07-12T15:15:51.481 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-07-12T15:15:51.840 INFO:teuthology.orchestra.run.smithi145.stdout:(22/23): ceph-common-18.1.2-55.gb1bd0540.el8.x8 3.7 MB/s | 22 MB 00:05 2023-07-12T15:15:51.869 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-07-12T15:15:52.080 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librados2-2:18.1.2-55.gb1bd0540.el8.x86_64 3/25 2023-07-12T15:15:52.176 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:18.1.2-55.gb1bd0540.el8.x86_64 3/25 2023-07-12T15:15:52.377 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-2:18.1.2-55.gb1bd0540.el8.x86_64 4/25 2023-07-12T15:15:52.540 INFO:teuthology.orchestra.run.smithi145.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 696 kB/s | 1.7 MB 00:02 2023-07-12T15:15:52.544 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:15:52.544 INFO:teuthology.orchestra.run.smithi145.stdout:Total 8.8 MB/s | 59 MB 00:06 2023-07-12T15:15:52.590 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:15:52.635 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:15:52.635 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:15:52.657 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2023-07-12T15:15:52.855 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-2:18.1.2-55.gb1bd0540.el8.x86_64 6/25 2023-07-12T15:15:52.966 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:15:52.970 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:15:53.268 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-2:18.1.2-55.gb1bd0540.el8.x86_64 6/25 2023-07-12T15:15:53.513 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librbd1-2:18.1.2-55.gb1bd0540.el8.x86_64 7/25 2023-07-12T15:15:53.603 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:18.1.2-55.gb1bd0540.el8.x86_64 7/25 2023-07-12T15:15:53.805 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-07-12T15:15:53.869 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-07-12T15:15:54.036 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/25 2023-07-12T15:15:54.179 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:15:54.198 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-argparse-2:18.1.2-55.gb1bd0540.el8.x8 10/25 2023-07-12T15:15:54.374 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-2:18.1.2-55.gb1bd0540.el8.x86_64 11/25 2023-07-12T15:15:54.408 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-07-12T15:15:54.597 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-2:18.1.2-55.gb1bd0540.el8.x86_64 12/25 2023-07-12T15:15:54.602 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-07-12T15:15:54.763 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libradosstriper1-2:18.1.2-55.gb1bd0540.el8.x86_64 13/25 2023-07-12T15:15:54.833 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libradosstriper1-2:18.1.2-55.gb1bd0540.el8.x86_64 13/25 2023-07-12T15:15:54.995 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-07-12T15:15:54.997 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2023-07-12T15:15:55.180 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2023-07-12T15:15:55.210 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librados2-2:18.1.2-55.gb1bd0540.el8.x86_64 3/25 2023-07-12T15:15:55.306 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-2:18.1.2-55.gb1bd0540.el8.x86_64 3/25 2023-07-12T15:15:55.650 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rados-2:18.1.2-55.gb1bd0540.el8.x86_64 4/25 2023-07-12T15:15:55.909 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2023-07-12T15:15:55.978 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2023-07-12T15:15:56.135 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs2-2:18.1.2-55.gb1bd0540.el8.x86_64 6/25 2023-07-12T15:15:56.196 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librgw2-2:18.1.2-55.gb1bd0540.el8.x86_64 17/25 2023-07-12T15:15:56.274 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librgw2-2:18.1.2-55.gb1bd0540.el8.x86_64 17/25 2023-07-12T15:15:56.489 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-2:18.1.2-55.gb1bd0540.el8.x86_64 18/25 2023-07-12T15:15:56.531 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephfs2-2:18.1.2-55.gb1bd0540.el8.x86_64 6/25 2023-07-12T15:15:56.655 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_ 19/25 2023-07-12T15:15:56.727 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librbd1-2:18.1.2-55.gb1bd0540.el8.x86_64 7/25 2023-07-12T15:15:56.821 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-2:18.1.2-55.gb1bd0540.el8.x86_64 7/25 2023-07-12T15:15:57.028 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-07-12T15:15:57.226 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-07-12T15:15:57.284 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-07-12T15:15:57.428 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-argparse-2:18.1.2-55.gb1bd0540.el8.x8 10/25 2023-07-12T15:15:57.641 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cephfs-2:18.1.2-55.gb1bd0540.el8.x86_64 11/25 2023-07-12T15:15:57.827 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rbd-2:18.1.2-55.gb1bd0540.el8.x86_64 12/25 2023-07-12T15:15:57.968 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libradosstriper1-2:18.1.2-55.gb1bd0540.el8.x86_64 13/25 2023-07-12T15:15:58.046 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libradosstriper1-2:18.1.2-55.gb1bd0540.el8.x86_64 13/25 2023-07-12T15:15:58.219 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2023-07-12T15:15:58.406 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2023-07-12T15:15:59.206 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2023-07-12T15:15:59.376 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librgw2-2:18.1.2-55.gb1bd0540.el8.x86_64 17/25 2023-07-12T15:15:59.452 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librgw2-2:18.1.2-55.gb1bd0540.el8.x86_64 17/25 2023-07-12T15:15:59.692 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rgw-2:18.1.2-55.gb1bd0540.el8.x86_64 18/25 2023-07-12T15:15:59.843 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_ 19/25 2023-07-12T15:16:00.263 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_64 20/25 2023-07-12T15:16:00.563 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_64 20/25 2023-07-12T15:16:01.152 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_64 20/25 2023-07-12T15:16:01.475 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-2:18.1.2-55.gb1bd0540.el8.x86_64 21/25 2023-07-12T15:16:01.560 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-2:18.1.2-55.gb1bd0540.el8.x86_64 21/25 2023-07-12T15:16:01.720 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-selinux-2:18.1.2-55.gb1bd0540.el8.x86_64 22/25 2023-07-12T15:16:03.852 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_64 20/25 2023-07-12T15:16:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_64 20/25 2023-07-12T15:16:04.729 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_64 20/25 2023-07-12T15:16:05.026 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-base-2:18.1.2-55.gb1bd0540.el8.x86_64 21/25 2023-07-12T15:16:05.104 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-base-2:18.1.2-55.gb1bd0540.el8.x86_64 21/25 2023-07-12T15:16:05.275 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-selinux-2:18.1.2-55.gb1bd0540.el8.x86_64 22/25 2023-07-12T15:16:20.308 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-selinux-2:18.1.2-55.gb1bd0540.el8.x86_64 22/25 2023-07-12T15:16:20.309 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /sys 2023-07-12T15:16:20.309 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /proc 2023-07-12T15:16:20.309 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /dev 2023-07-12T15:16:20.309 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /run 2023-07-12T15:16:20.309 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /mnt 2023-07-12T15:16:20.309 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /var/tmp 2023-07-12T15:16:20.310 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /home 2023-07-12T15:16:20.310 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /tmp 2023-07-12T15:16:20.310 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /dev 2023-07-12T15:16:20.310 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:16:21.621 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-2:18.1.2-55.gb1bd0540.el8.x86_64 23/25 2023-07-12T15:16:21.641 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-2:18.1.2-55.gb1bd0540.el8.x86_64 23/25 2023-07-12T15:16:21.641 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-07-12T15:16:21.836 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-07-12T15:16:21.836 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-07-12T15:16:22.446 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-07-12T15:16:22.447 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-2:18.1.2-55.gb1bd0540.el8.x86_64 1/25 2023-07-12T15:16:22.447 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_64 2/25 2023-07-12T15:16:22.447 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-2:18.1.2-55.gb1bd0540.el8.x86_64 3/25 2023-07-12T15:16:22.447 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-selinux-2:18.1.2-55.gb1bd0540.el8.x86_64 4/25 2023-07-12T15:16:22.447 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-2:18.1.2-55.gb1bd0540.el8.x86_64 5/25 2023-07-12T15:16:22.447 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libradosstriper1-2:18.1.2-55.gb1bd0540.el8.x86_64 6/25 2023-07-12T15:16:22.448 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librgw2-2:18.1.2-55.gb1bd0540.el8.x86_64 7/25 2023-07-12T15:16:22.448 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-argparse-2:18.1.2-55.gb1bd0540.el8.x8 8/25 2023-07-12T15:16:22.448 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_ 9/25 2023-07-12T15:16:22.448 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-2:18.1.2-55.gb1bd0540.el8.x86_64 10/25 2023-07-12T15:16:22.448 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-2:18.1.2-55.gb1bd0540.el8.x86_64 11/25 2023-07-12T15:16:22.448 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-2:18.1.2-55.gb1bd0540.el8.x86_64 12/25 2023-07-12T15:16:22.449 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-2:18.1.2-55.gb1bd0540.el8.x86_64 13/25 2023-07-12T15:16:22.449 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-07-12T15:16:22.449 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-07-12T15:16:22.449 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-07-12T15:16:22.449 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-07-12T15:16:22.449 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-07-12T15:16:22.449 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-07-12T15:16:22.450 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2023-07-12T15:16:22.450 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2023-07-12T15:16:22.450 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:18.1.2-55.gb1bd0540.el8.x86_64 22/25 2023-07-12T15:16:22.450 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-07-12T15:16:22.450 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:18.1.2-55.gb1bd0540.el8.x86_64 24/25 2023-07-12T15:16:23.789 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-07-12T15:16:23.790 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:50 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:16:24.004 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-selinux-2:18.1.2-55.gb1bd0540.el8.x86_64 22/25 2023-07-12T15:16:24.005 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /sys 2023-07-12T15:16:24.005 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /proc 2023-07-12T15:16:24.005 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /dev 2023-07-12T15:16:24.005 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /run 2023-07-12T15:16:24.006 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /mnt 2023-07-12T15:16:24.006 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /var/tmp 2023-07-12T15:16:24.006 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /home 2023-07-12T15:16:24.006 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /tmp 2023-07-12T15:16:24.007 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /dev 2023-07-12T15:16:24.007 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:16:25.355 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-radosgw-2:18.1.2-55.gb1bd0540.el8.x86_64 23/25 2023-07-12T15:16:25.376 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-radosgw-2:18.1.2-55.gb1bd0540.el8.x86_64 23/25 2023-07-12T15:16:25.376 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-07-12T15:16:25.565 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-07-12T15:16:25.565 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-07-12T15:16:26.149 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:16:26.195 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-07-12T15:16:26.195 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-base-2:18.1.2-55.gb1bd0540.el8.x86_64 1/25 2023-07-12T15:16:26.196 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_64 2/25 2023-07-12T15:16:26.196 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-radosgw-2:18.1.2-55.gb1bd0540.el8.x86_64 3/25 2023-07-12T15:16:26.196 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-selinux-2:18.1.2-55.gb1bd0540.el8.x86_64 4/25 2023-07-12T15:16:26.196 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs2-2:18.1.2-55.gb1bd0540.el8.x86_64 5/25 2023-07-12T15:16:26.196 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libradosstriper1-2:18.1.2-55.gb1bd0540.el8.x86_64 6/25 2023-07-12T15:16:26.196 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librgw2-2:18.1.2-55.gb1bd0540.el8.x86_64 7/25 2023-07-12T15:16:26.197 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-argparse-2:18.1.2-55.gb1bd0540.el8.x8 8/25 2023-07-12T15:16:26.197 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_ 9/25 2023-07-12T15:16:26.197 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cephfs-2:18.1.2-55.gb1bd0540.el8.x86_64 10/25 2023-07-12T15:16:26.197 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rados-2:18.1.2-55.gb1bd0540.el8.x86_64 11/25 2023-07-12T15:16:26.197 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rbd-2:18.1.2-55.gb1bd0540.el8.x86_64 12/25 2023-07-12T15:16:26.197 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rgw-2:18.1.2-55.gb1bd0540.el8.x86_64 13/25 2023-07-12T15:16:26.197 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-07-12T15:16:26.198 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-07-12T15:16:26.198 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-07-12T15:16:26.198 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-07-12T15:16:26.198 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 18/25 2023-07-12T15:16:26.198 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2023-07-12T15:16:26.198 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 20/25 2023-07-12T15:16:26.199 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2023-07-12T15:16:26.199 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-2:18.1.2-55.gb1bd0540.el8.x86_64 22/25 2023-07-12T15:16:26.199 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-07-12T15:16:26.199 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-2:18.1.2-55.gb1bd0540.el8.x86_64 24/25 2023-07-12T15:16:26.337 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:16:26.337 INFO:teuthology.orchestra.run.smithi069.stdout:Upgraded: 2023-07-12T15:16:26.337 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.337 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.337 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:16:26.338 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.338 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.338 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.338 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.338 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-12T15:16:26.338 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.338 INFO:teuthology.orchestra.run.smithi069.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-12T15:16:26.338 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-12T15:16:26.339 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.339 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-12T15:16:26.339 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.339 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-12T15:16:26.339 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-12T15:16:26.339 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.339 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.340 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.340 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-12T15:16:26.340 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.340 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.340 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:26.340 INFO:teuthology.orchestra.run.smithi069.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-12T15:16:26.340 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:16:26.341 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:16:26.527 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-test 2023-07-12T15:16:26.873 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:16:27.596 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-07-12T15:16:27.596 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:53 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:16:27.673 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:54 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:16:29.051 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:16:29.052 INFO:teuthology.orchestra.run.smithi069.stdout:========================================================================================== 2023-07-12T15:16:29.053 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-07-12T15:16:29.053 INFO:teuthology.orchestra.run.smithi069.stdout:========================================================================================== 2023-07-12T15:16:29.053 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:16:29.053 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 47 M 2023-07-12T15:16:29.053 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-07-12T15:16:29.053 INFO:teuthology.orchestra.run.smithi069.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-12T15:16:29.054 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 137 k 2023-07-12T15:16:29.054 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-12T15:16:29.054 INFO:teuthology.orchestra.run.smithi069.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-12T15:16:29.054 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-12T15:16:29.054 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:16:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:16:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:========================================================================================== 2023-07-12T15:16:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:Install 6 Packages 2023-07-12T15:16:29.056 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:16:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 48 M 2023-07-12T15:16:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 211 M 2023-07-12T15:16:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:16:29.100 INFO:teuthology.orchestra.run.smithi069.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.5 MB/s | 70 kB 00:00 2023-07-12T15:16:29.192 INFO:teuthology.orchestra.run.smithi069.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2023-07-12T15:16:29.284 INFO:teuthology.orchestra.run.smithi069.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2023-07-12T15:16:29.318 INFO:teuthology.orchestra.run.smithi069.stdout:(4/6): libcephsqlite-18.1.2-55.gb1bd0540.el8.x8 521 kB/s | 137 kB 00:00 2023-07-12T15:16:29.343 INFO:teuthology.orchestra.run.smithi069.stdout:(5/6): jq-1.6-3.el8.x86_64.rpm 3.3 MB/s | 202 kB 00:00 2023-07-12T15:16:30.057 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:16:30.246 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:16:30.246 INFO:teuthology.orchestra.run.smithi145.stdout:Upgraded: 2023-07-12T15:16:30.246 INFO:teuthology.orchestra.run.smithi145.stdout: librados2-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.247 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.247 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:16:30.247 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.247 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.247 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.247 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.248 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-12T15:16:30.248 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.248 INFO:teuthology.orchestra.run.smithi145.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-12T15:16:30.248 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-12T15:16:30.248 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.248 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-12T15:16:30.248 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.249 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-12T15:16:30.249 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-12T15:16:30.249 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.249 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.249 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.249 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-12T15:16:30.249 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.249 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.250 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:16:30.250 INFO:teuthology.orchestra.run.smithi145.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-12T15:16:30.250 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:16:30.250 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:16:30.422 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-test 2023-07-12T15:16:30.770 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:16:31.293 INFO:teuthology.orchestra.run.smithi069.stdout:(6/6): ceph-test-18.1.2-55.gb1bd0540.el8.x86_64 21 MB/s | 47 MB 00:02 2023-07-12T15:16:31.295 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:16:31.296 INFO:teuthology.orchestra.run.smithi069.stdout:Total 21 MB/s | 48 MB 00:02 2023-07-12T15:16:31.332 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:16:31.353 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:16:31.354 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:16:31.607 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:16:31.611 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:16:31.681 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:57 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:16:32.071 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:16:32.249 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2023-07-12T15:16:32.317 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2023-07-12T15:16:32.501 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : jq-1.6-3.el8.x86_64 2/6 2023-07-12T15:16:32.683 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2023-07-12T15:16:32.819 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-07-12T15:16:33.024 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephsqlite-2:18.1.2-55.gb1bd0540.el8.x86_64 5/6 2023-07-12T15:16:33.137 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:16:33.138 INFO:teuthology.orchestra.run.smithi145.stdout:========================================================================================== 2023-07-12T15:16:33.138 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2023-07-12T15:16:33.139 INFO:teuthology.orchestra.run.smithi145.stdout:========================================================================================== 2023-07-12T15:16:33.139 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:16:33.139 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 47 M 2023-07-12T15:16:33.140 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2023-07-12T15:16:33.140 INFO:teuthology.orchestra.run.smithi145.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-12T15:16:33.140 INFO:teuthology.orchestra.run.smithi145.stdout: libcephsqlite x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 137 k 2023-07-12T15:16:33.140 INFO:teuthology.orchestra.run.smithi145.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-12T15:16:33.140 INFO:teuthology.orchestra.run.smithi145.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-12T15:16:33.140 INFO:teuthology.orchestra.run.smithi145.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-12T15:16:33.141 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:16:33.141 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:16:33.141 INFO:teuthology.orchestra.run.smithi145.stdout:========================================================================================== 2023-07-12T15:16:33.141 INFO:teuthology.orchestra.run.smithi145.stdout:Install 6 Packages 2023-07-12T15:16:33.141 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:16:33.142 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 48 M 2023-07-12T15:16:33.143 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 211 M 2023-07-12T15:16:33.143 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:16:33.188 INFO:teuthology.orchestra.run.smithi145.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-07-12T15:16:33.413 INFO:teuthology.orchestra.run.smithi145.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.4 MB/s | 323 kB 00:00 2023-07-12T15:16:33.546 INFO:teuthology.orchestra.run.smithi145.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2023-07-12T15:16:33.630 INFO:teuthology.orchestra.run.smithi145.stdout:(4/6): jq-1.6-3.el8.x86_64.rpm 2.4 MB/s | 202 kB 00:00 2023-07-12T15:16:33.722 INFO:teuthology.orchestra.run.smithi145.stdout:(5/6): libcephsqlite-18.1.2-55.gb1bd0540.el8.x8 236 kB/s | 137 kB 00:00 2023-07-12T15:16:37.488 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephsqlite-2:18.1.2-55.gb1bd0540.el8.x86_64 5/6 2023-07-12T15:16:37.568 INFO:teuthology.orchestra.run.smithi145.stdout:(6/6): ceph-test-18.1.2-55.gb1bd0540.el8.x86_64 11 MB/s | 47 MB 00:04 2023-07-12T15:16:37.570 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:16:37.570 INFO:teuthology.orchestra.run.smithi145.stdout:Total 11 MB/s | 48 MB 00:04 2023-07-12T15:16:37.607 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:16:37.628 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:16:37.628 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:16:37.770 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-2:18.1.2-55.gb1bd0540.el8.x86_64 6/6 2023-07-12T15:16:37.882 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:16:37.886 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:16:37.994 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-test-2:18.1.2-55.gb1bd0540.el8.x86_64 6/6 2023-07-12T15:16:37.994 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-2:18.1.2-55.gb1bd0540.el8.x86_64 1/6 2023-07-12T15:16:37.994 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephsqlite-2:18.1.2-55.gb1bd0540.el8.x86_64 2/6 2023-07-12T15:16:37.994 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-07-12T15:16:37.994 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2023-07-12T15:16:37.995 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-07-12T15:16:38.333 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:16:38.469 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2023-07-12T15:16:38.552 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2023-07-12T15:16:38.780 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : jq-1.6-3.el8.x86_64 2/6 2023-07-12T15:16:38.969 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2023-07-12T15:16:39.114 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-07-12T15:16:39.138 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : jq-1.6-3.el8.x86_64 6/6 2023-07-12T15:16:39.138 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:06 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:16:39.294 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephsqlite-2:18.1.2-55.gb1bd0540.el8.x86_64 5/6 2023-07-12T15:16:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephsqlite-2:18.1.2-55.gb1bd0540.el8.x86_64 5/6 2023-07-12T15:16:43.882 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:16:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-test-2:18.1.2-55.gb1bd0540.el8.x86_64 6/6 2023-07-12T15:16:44.067 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:16:44.068 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:16:44.068 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-2:18.1.2-55.gb1bd0540.el8.x86_64 jq-1.6-3.el8.x86_64 2023-07-12T15:16:44.068 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite-2:18.1.2-55.gb1bd0540.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-07-12T15:16:44.068 INFO:teuthology.orchestra.run.smithi069.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-07-12T15:16:44.068 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:16:44.068 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:16:44.234 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph 2023-07-12T15:16:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-test-2:18.1.2-55.gb1bd0540.el8.x86_64 6/6 2023-07-12T15:16:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-test-2:18.1.2-55.gb1bd0540.el8.x86_64 1/6 2023-07-12T15:16:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephsqlite-2:18.1.2-55.gb1bd0540.el8.x86_64 2/6 2023-07-12T15:16:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-07-12T15:16:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2023-07-12T15:16:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-07-12T15:16:44.566 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:16:45.387 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : jq-1.6-3.el8.x86_64 6/6 2023-07-12T15:16:45.388 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:11 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:16:45.391 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:12 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:16:46.773 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:16:46.777 INFO:teuthology.orchestra.run.smithi069.stdout:===================================================================================================== 2023-07-12T15:16:46.777 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-07-12T15:16:46.777 INFO:teuthology.orchestra.run.smithi069.stdout:===================================================================================================== 2023-07-12T15:16:46.777 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:16:46.777 INFO:teuthology.orchestra.run.smithi069.stdout: ceph x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 6.4 k 2023-07-12T15:16:46.778 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-07-12T15:16:46.778 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 2.3 M 2023-07-12T15:16:46.778 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 1.5 M 2023-07-12T15:16:46.778 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 240 k 2023-07-12T15:16:46.779 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 4.6 M 2023-07-12T15:16:46.779 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 18 M 2023-07-12T15:16:46.779 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-12T15:16:46.780 INFO:teuthology.orchestra.run.smithi069.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-12T15:16:46.780 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-12T15:16:46.780 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-12T15:16:46.780 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-12T15:16:46.781 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-12T15:16:46.781 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-12T15:16:46.781 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-12T15:16:46.782 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-12T15:16:46.782 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-12T15:16:46.782 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-12T15:16:46.782 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-12T15:16:46.783 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-12T15:16:46.783 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-12T15:16:46.783 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-12T15:16:46.783 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-12T15:16:46.784 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-12T15:16:46.784 INFO:teuthology.orchestra.run.smithi069.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-12T15:16:46.784 INFO:teuthology.orchestra.run.smithi069.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-12T15:16:46.784 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-12T15:16:46.784 INFO:teuthology.orchestra.run.smithi069.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-12T15:16:46.784 INFO:teuthology.orchestra.run.smithi069.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-12T15:16:46.785 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-12T15:16:46.785 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-12T15:16:46.785 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-12T15:16:46.785 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-12T15:16:46.785 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:16:46.785 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:16:46.785 INFO:teuthology.orchestra.run.smithi069.stdout:===================================================================================================== 2023-07-12T15:16:46.786 INFO:teuthology.orchestra.run.smithi069.stdout:Install 32 Packages 2023-07-12T15:16:46.786 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:16:46.787 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 30 M 2023-07-12T15:16:46.787 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 119 M 2023-07-12T15:16:46.787 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:16:47.026 INFO:teuthology.orchestra.run.smithi069.stdout:(1/32): ceph-18.1.2-55.gb1bd0540.el8.x86_64.rpm 30 kB/s | 6.4 kB 00:00 2023-07-12T15:16:47.234 INFO:teuthology.orchestra.run.smithi069.stdout:(2/32): ceph-mds-18.1.2-55.gb1bd0540.el8.x86_64 5.4 MB/s | 2.3 MB 00:00 2023-07-12T15:16:47.293 INFO:teuthology.orchestra.run.smithi069.stdout:(3/32): ceph-mgr-18.1.2-55.gb1bd0540.el8.x86_64 3.2 MB/s | 1.5 MB 00:00 2023-07-12T15:16:47.376 INFO:teuthology.orchestra.run.smithi069.stdout:(4/32): ceph-mgr-modules-core-18.1.2-55.gb1bd05 2.8 MB/s | 240 kB 00:00 2023-07-12T15:16:47.685 INFO:teuthology.orchestra.run.smithi069.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 141 kB/s | 44 kB 00:00 2023-07-12T15:16:47.902 INFO:teuthology.orchestra.run.smithi069.stdout:(6/32): ceph-mon-18.1.2-55.gb1bd0540.el8.x86_64 5.3 MB/s | 4.6 MB 00:00 2023-07-12T15:16:48.186 INFO:teuthology.orchestra.run.smithi069.stdout:(7/32): ceph-osd-18.1.2-55.gb1bd0540.el8.x86_64 19 MB/s | 18 MB 00:00 2023-07-12T15:16:48.486 INFO:teuthology.orchestra.run.smithi069.stdout:(8/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 231 kB/s | 185 kB 00:00 2023-07-12T15:16:48.528 INFO:teuthology.orchestra.run.smithi069.stdout:(9/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 276 kB/s | 173 kB 00:00 2023-07-12T15:16:48.561 INFO:teuthology.orchestra.run.smithi069.stdout:(10/32): python3-jaraco-6.2-6.el8.noarch.rpm 147 kB/s | 11 kB 00:00 2023-07-12T15:16:48.595 INFO:teuthology.orchestra.run.smithi069.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 262 kB/s | 18 kB 00:00 2023-07-12T15:16:48.679 INFO:teuthology.orchestra.run.smithi069.stdout:(12/32): python3-more-itertools-7.2.0-3.el8.noa 706 kB/s | 59 kB 00:00 2023-07-12T15:16:48.704 INFO:teuthology.orchestra.run.smithi069.stdout:(13/32): python3-cherrypy-18.4.0-1.el8.noarch.r 743 kB/s | 384 kB 00:00 2023-07-12T15:16:48.738 INFO:teuthology.orchestra.run.smithi069.stdout:(14/32): python3-logutils-0.3.5-11.el8.noarch.r 277 kB/s | 49 kB 00:00 2023-07-12T15:16:48.780 INFO:teuthology.orchestra.run.smithi069.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 207 kB/s | 16 kB 00:00 2023-07-12T15:16:48.805 INFO:teuthology.orchestra.run.smithi069.stdout:(16/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.2 MB/s | 283 kB 00:00 2023-07-12T15:16:48.830 INFO:teuthology.orchestra.run.smithi069.stdout:(17/32): python3-simplegeneric-0.8.1-17.el8.noa 203 kB/s | 19 kB 00:00 2023-07-12T15:16:48.855 INFO:teuthology.orchestra.run.smithi069.stdout:(18/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2023-07-12T15:16:48.881 INFO:teuthology.orchestra.run.smithi069.stdout:(19/32): python3-tempora-1.14.1-5.el8.noarch.rp 387 kB/s | 29 kB 00:00 2023-07-12T15:16:48.931 INFO:teuthology.orchestra.run.smithi069.stdout:(20/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-07-12T15:16:49.015 INFO:teuthology.orchestra.run.smithi069.stdout:(21/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2023-07-12T15:16:49.073 INFO:teuthology.orchestra.run.smithi069.stdout:(22/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2023-07-12T15:16:49.098 INFO:teuthology.orchestra.run.smithi069.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 275 kB/s | 23 kB 00:00 2023-07-12T15:16:49.124 INFO:teuthology.orchestra.run.smithi069.stdout:(24/32): python3-trustme-0.6.0-4.el8.noarch.rpm 94 kB/s | 27 kB 00:00 2023-07-12T15:16:49.291 INFO:teuthology.orchestra.run.smithi069.stdout:(25/32): python3-cryptography-3.2.1-5.el8.x86_6 2.8 MB/s | 559 kB 00:00 2023-07-12T15:16:49.324 INFO:teuthology.orchestra.run.smithi069.stdout:(26/32): python3-pycparser-2.14-14.el8.noarch.r 435 kB/s | 109 kB 00:00 2023-07-12T15:16:49.350 INFO:teuthology.orchestra.run.smithi069.stdout:(27/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.0 MB/s | 238 kB 00:00 2023-07-12T15:16:49.475 INFO:teuthology.orchestra.run.smithi069.stdout:(28/32): python3-markupsafe-0.23-19.el8.x86_64. 259 kB/s | 39 kB 00:00 2023-07-12T15:16:49.500 INFO:teuthology.orchestra.run.smithi069.stdout:(29/32): python3-pytz-2017.2-9.el8.noarch.rpm 259 kB/s | 54 kB 00:00 2023-07-12T15:16:49.525 INFO:teuthology.orchestra.run.smithi069.stdout:(30/32): python3-mako-1.0.6-13.el8.noarch.rpm 895 kB/s | 157 kB 00:00 2023-07-12T15:16:49.659 INFO:teuthology.orchestra.run.smithi069.stdout:(31/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.4 MB/s | 457 kB 00:00 2023-07-12T15:16:49.767 INFO:teuthology.orchestra.run.smithi069.stdout:(32/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 388 kB/s | 103 kB 00:00 2023-07-12T15:16:49.771 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:16:49.772 INFO:teuthology.orchestra.run.smithi069.stdout:Total 10 MB/s | 30 MB 00:02 2023-07-12T15:16:49.862 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:16:49.900 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:16:49.900 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:16:50.109 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:16:50.113 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:16:50.196 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:16:50.386 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:16:50.387 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:16:50.387 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test-2:18.1.2-55.gb1bd0540.el8.x86_64 jq-1.6-3.el8.x86_64 2023-07-12T15:16:50.387 INFO:teuthology.orchestra.run.smithi145.stdout: libcephsqlite-2:18.1.2-55.gb1bd0540.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-07-12T15:16:50.387 INFO:teuthology.orchestra.run.smithi145.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-07-12T15:16:50.387 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:16:50.387 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:16:50.524 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph 2023-07-12T15:16:50.859 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:16:51.555 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:16:51.776 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-07-12T15:16:52.079 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-07-12T15:16:52.283 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 3/32 2023-07-12T15:16:52.314 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:18 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:16:52.539 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2023-07-12T15:16:52.794 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/32 2023-07-12T15:16:53.001 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 6/32 2023-07-12T15:16:53.256 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2023-07-12T15:16:53.616 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 8/32 2023-07-12T15:16:53.777 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:16:53.781 INFO:teuthology.orchestra.run.smithi145.stdout:===================================================================================================== 2023-07-12T15:16:53.781 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2023-07-12T15:16:53.781 INFO:teuthology.orchestra.run.smithi145.stdout:===================================================================================================== 2023-07-12T15:16:53.781 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:16:53.781 INFO:teuthology.orchestra.run.smithi145.stdout: ceph x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 6.4 k 2023-07-12T15:16:53.781 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2023-07-12T15:16:53.782 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 2.3 M 2023-07-12T15:16:53.782 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 1.5 M 2023-07-12T15:16:53.782 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 240 k 2023-07-12T15:16:53.783 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 4.6 M 2023-07-12T15:16:53.783 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 18 M 2023-07-12T15:16:53.783 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-12T15:16:53.784 INFO:teuthology.orchestra.run.smithi145.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-12T15:16:53.784 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-12T15:16:53.784 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-12T15:16:53.785 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-12T15:16:53.785 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-12T15:16:53.785 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-12T15:16:53.786 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-12T15:16:53.786 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-12T15:16:53.786 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-12T15:16:53.787 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-12T15:16:53.787 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-12T15:16:53.788 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-12T15:16:53.788 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-12T15:16:53.788 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-12T15:16:53.789 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-12T15:16:53.789 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-12T15:16:53.789 INFO:teuthology.orchestra.run.smithi145.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-12T15:16:53.789 INFO:teuthology.orchestra.run.smithi145.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-12T15:16:53.790 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-12T15:16:53.790 INFO:teuthology.orchestra.run.smithi145.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-12T15:16:53.790 INFO:teuthology.orchestra.run.smithi145.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-12T15:16:53.791 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-12T15:16:53.791 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-12T15:16:53.791 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-12T15:16:53.791 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-12T15:16:53.792 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:16:53.792 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:16:53.792 INFO:teuthology.orchestra.run.smithi145.stdout:===================================================================================================== 2023-07-12T15:16:53.792 INFO:teuthology.orchestra.run.smithi145.stdout:Install 32 Packages 2023-07-12T15:16:53.793 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:16:53.794 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 30 M 2023-07-12T15:16:53.794 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 119 M 2023-07-12T15:16:53.795 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:16:53.882 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 9/32 2023-07-12T15:16:53.971 INFO:teuthology.orchestra.run.smithi145.stdout:(1/32): ceph-18.1.2-55.gb1bd0540.el8.x86_64.rpm 52 kB/s | 6.4 kB 00:00 2023-07-12T15:16:54.095 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2023-07-12T15:16:54.155 INFO:teuthology.orchestra.run.smithi145.stdout:(2/32): ceph-mds-18.1.2-55.gb1bd0540.el8.x86_64 7.4 MB/s | 2.3 MB 00:00 2023-07-12T15:16:54.297 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2023-07-12T15:16:54.537 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2023-07-12T15:16:54.765 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2023-07-12T15:16:54.780 INFO:teuthology.orchestra.run.smithi145.stdout:(3/32): ceph-mon-18.1.2-55.gb1bd0540.el8.x86_64 5.7 MB/s | 4.6 MB 00:00 2023-07-12T15:16:54.894 INFO:teuthology.orchestra.run.smithi145.stdout:(4/32): ceph-mgr-modules-core-18.1.2-55.gb1bd05 2.1 MB/s | 240 kB 00:00 2023-07-12T15:16:54.986 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2023-07-12T15:16:55.139 INFO:teuthology.orchestra.run.smithi145.stdout:(5/32): ceph-osd-18.1.2-55.gb1bd0540.el8.x86_64 18 MB/s | 18 MB 00:00 2023-07-12T15:16:55.141 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2023-07-12T15:16:55.181 INFO:teuthology.orchestra.run.smithi145.stdout:(6/32): ceph-mgr-18.1.2-55.gb1bd0540.el8.x86_64 1.2 MB/s | 1.5 MB 00:01 2023-07-12T15:16:55.298 INFO:teuthology.orchestra.run.smithi145.stdout:(7/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 108 kB/s | 44 kB 00:00 2023-07-12T15:16:55.365 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2023-07-12T15:16:55.423 INFO:teuthology.orchestra.run.smithi145.stdout:(8/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 654 kB/s | 185 kB 00:00 2023-07-12T15:16:55.490 INFO:teuthology.orchestra.run.smithi145.stdout:(9/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 559 kB/s | 173 kB 00:00 2023-07-12T15:16:55.515 INFO:teuthology.orchestra.run.smithi145.stdout:(10/32): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2023-07-12T15:16:55.540 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2023-07-12T15:16:55.599 INFO:teuthology.orchestra.run.smithi145.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2023-07-12T15:16:55.624 INFO:teuthology.orchestra.run.smithi145.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 448 kB/s | 49 kB 00:00 2023-07-12T15:16:55.683 INFO:teuthology.orchestra.run.smithi145.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 703 kB/s | 59 kB 00:00 2023-07-12T15:16:55.710 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2023-07-12T15:16:55.758 INFO:teuthology.orchestra.run.smithi145.stdout:(14/32): python3-portend-2.6-1.el8.noarch.rpm 208 kB/s | 16 kB 00:00 2023-07-12T15:16:55.825 INFO:teuthology.orchestra.run.smithi145.stdout:(15/32): python3-cherrypy-18.4.0-1.el8.noarch.r 731 kB/s | 384 kB 00:00 2023-07-12T15:16:55.850 INFO:teuthology.orchestra.run.smithi145.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 204 kB/s | 19 kB 00:00 2023-07-12T15:16:55.876 INFO:teuthology.orchestra.run.smithi145.stdout:(17/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2023-07-12T15:16:55.916 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2023-07-12T15:16:55.959 INFO:teuthology.orchestra.run.smithi145.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 269 kB/s | 29 kB 00:00 2023-07-12T15:16:55.984 INFO:teuthology.orchestra.run.smithi145.stdout:(19/32): python3-singledispatch-3.4.0.3-18.el8. 178 kB/s | 24 kB 00:00 2023-07-12T15:16:56.010 INFO:teuthology.orchestra.run.smithi145.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 206 kB/s | 27 kB 00:00 2023-07-12T15:16:56.102 INFO:teuthology.orchestra.run.smithi145.stdout:(21/32): python3-webtest-2.0.33-1.el8.noarch.rp 933 kB/s | 85 kB 00:00 2023-07-12T15:16:56.135 INFO:teuthology.orchestra.run.smithi145.stdout:(22/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.4 MB/s | 255 kB 00:00 2023-07-12T15:16:56.161 INFO:teuthology.orchestra.run.smithi145.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 391 kB/s | 23 kB 00:00 2023-07-12T15:16:56.219 INFO:teuthology.orchestra.run.smithi145.stdout:(24/32): python3-pytz-2017.2-9.el8.noarch.rpm 649 kB/s | 54 kB 00:00 2023-07-12T15:16:56.270 INFO:teuthology.orchestra.run.smithi145.stdout:(25/32): python3-markupsafe-0.23-19.el8.x86_64. 377 kB/s | 39 kB 00:00 2023-07-12T15:16:56.295 INFO:teuthology.orchestra.run.smithi145.stdout:(26/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.0 MB/s | 157 kB 00:00 2023-07-12T15:16:56.328 INFO:teuthology.orchestra.run.smithi145.stdout:(27/32): python3-werkzeug-0.12.2-4.el8.noarch.r 7.6 MB/s | 457 kB 00:00 2023-07-12T15:16:56.354 INFO:teuthology.orchestra.run.smithi145.stdout:(28/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 1.7 MB/s | 103 kB 00:00 2023-07-12T15:16:56.379 INFO:teuthology.orchestra.run.smithi145.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 680 kB/s | 251 kB 00:00 2023-07-12T15:16:56.399 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2023-07-12T15:16:56.404 INFO:teuthology.orchestra.run.smithi145.stdout:(30/32): python3-pycparser-2.14-14.el8.noarch.r 1.4 MB/s | 109 kB 00:00 2023-07-12T15:16:56.438 INFO:teuthology.orchestra.run.smithi145.stdout:(31/32): python3-cryptography-3.2.1-5.el8.x86_6 6.5 MB/s | 559 kB 00:00 2023-07-12T15:16:56.463 INFO:teuthology.orchestra.run.smithi145.stdout:(32/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.8 MB/s | 238 kB 00:00 2023-07-12T15:16:56.468 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:16:56.468 INFO:teuthology.orchestra.run.smithi145.stdout:Total 11 MB/s | 30 MB 00:02 2023-07-12T15:16:56.557 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:16:56.594 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:16:56.594 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:16:56.634 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-07-12T15:16:56.804 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:16:56.807 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:16:56.923 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-07-12T15:16:57.226 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-07-12T15:16:57.396 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2023-07-12T15:16:57.707 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2023-07-12T15:16:57.968 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:16:58.034 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-07-12T15:16:58.162 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-07-12T15:16:58.402 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-07-12T15:16:58.412 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-modules-core-2:18.1.2-55.gb1bd0540.el8.no 27/32 2023-07-12T15:16:58.583 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-2:18.1.2-55.gb1bd0540.el8.x86_64 28/32 2023-07-12T15:16:58.600 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 3/32 2023-07-12T15:16:58.956 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 4/32 2023-07-12T15:16:59.654 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 5/32 2023-07-12T15:16:59.841 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 6/32 2023-07-12T15:17:00.010 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 7/32 2023-07-12T15:17:00.154 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-2:18.1.2-55.gb1bd0540.el8.x86_64 28/32 2023-07-12T15:17:00.295 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 8/32 2023-07-12T15:17:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-2:18.1.2-55.gb1bd0540.el8.x86_64 29/32 2023-07-12T15:17:00.544 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 9/32 2023-07-12T15:17:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 10/32 2023-07-12T15:17:00.841 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-2:18.1.2-55.gb1bd0540.el8.x86_64 29/32 2023-07-12T15:17:01.012 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-2:18.1.2-55.gb1bd0540.el8.x86_64 30/32 2023-07-12T15:17:01.037 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 11/32 2023-07-12T15:17:01.226 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-2:18.1.2-55.gb1bd0540.el8.x86_64 30/32 2023-07-12T15:17:01.264 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 12/32 2023-07-12T15:17:01.403 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-2:18.1.2-55.gb1bd0540.el8.x86_64 31/32 2023-07-12T15:17:01.421 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-2:18.1.2-55.gb1bd0540.el8.x86_64 31/32 2023-07-12T15:17:01.477 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2023-07-12T15:17:01.526 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-2:18.1.2-55.gb1bd0540.el8.x86_64 32/32 2023-07-12T15:17:01.680 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2023-07-12T15:17:01.819 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2023-07-12T15:17:02.012 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-2:18.1.2-55.gb1bd0540.el8.x86_64 32/32 2023-07-12T15:17:02.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-2:18.1.2-55.gb1bd0540.el8.x86_64 1/32 2023-07-12T15:17:02.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-2:18.1.2-55.gb1bd0540.el8.x86_64 2/32 2023-07-12T15:17:02.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-2:18.1.2-55.gb1bd0540.el8.x86_64 3/32 2023-07-12T15:17:02.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-2:18.1.2-55.gb1bd0540.el8.x86_64 4/32 2023-07-12T15:17:02.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-2:18.1.2-55.gb1bd0540.el8.x86_64 5/32 2023-07-12T15:17:02.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-modules-core-2:18.1.2-55.gb1bd0540.el8.no 6/32 2023-07-12T15:17:02.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2023-07-12T15:17:02.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2023-07-12T15:17:02.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2023-07-12T15:17:02.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2023-07-12T15:17:02.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2023-07-12T15:17:02.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2023-07-12T15:17:02.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2023-07-12T15:17:02.015 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2023-07-12T15:17:02.015 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2023-07-12T15:17:02.015 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2023-07-12T15:17:02.015 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2023-07-12T15:17:02.015 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2023-07-12T15:17:02.015 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2023-07-12T15:17:02.015 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2023-07-12T15:17:02.015 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2023-07-12T15:17:02.016 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2023-07-12T15:17:02.016 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2023-07-12T15:17:02.016 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2023-07-12T15:17:02.016 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2023-07-12T15:17:02.016 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 26/32 2023-07-12T15:17:02.016 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 27/32 2023-07-12T15:17:02.016 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/32 2023-07-12T15:17:02.017 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2023-07-12T15:17:02.017 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 30/32 2023-07-12T15:17:02.017 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 31/32 2023-07-12T15:17:02.018 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2023-07-12T15:17:02.217 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2023-07-12T15:17:02.354 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2023-07-12T15:17:02.601 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2023-07-12T15:17:02.826 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2023-07-12T15:17:03.062 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-07-12T15:17:03.367 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-07-12T15:17:03.462 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 32/32 2023-07-12T15:17:03.462 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:30 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:17:03.670 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-07-12T15:17:03.848 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2023-07-12T15:17:04.186 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2023-07-12T15:17:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-07-12T15:17:05.184 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-modules-core-2:18.1.2-55.gb1bd0540.el8.no 27/32 2023-07-12T15:17:05.323 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-2:18.1.2-55.gb1bd0540.el8.x86_64 28/32 2023-07-12T15:17:05.779 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:17:05.969 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:17:05.970 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:17:05.970 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:17:05.970 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:17:05.970 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:17:05.971 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:17:05.971 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:17:05.971 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:17:05.972 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-12T15:17:05.972 INFO:teuthology.orchestra.run.smithi069.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-12T15:17:05.972 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-12T15:17:05.972 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-12T15:17:05.973 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-12T15:17:05.973 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-12T15:17:05.973 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-12T15:17:05.973 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-12T15:17:05.973 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-12T15:17:05.974 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-12T15:17:05.974 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-12T15:17:05.974 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-12T15:17:05.975 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-12T15:17:05.975 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend-2.6-1.el8.noarch 2023-07-12T15:17:05.975 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-12T15:17:05.975 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-12T15:17:05.975 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-12T15:17:05.976 INFO:teuthology.orchestra.run.smithi069.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-12T15:17:05.976 INFO:teuthology.orchestra.run.smithi069.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-12T15:17:05.976 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-12T15:17:05.976 INFO:teuthology.orchestra.run.smithi069.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-12T15:17:05.976 INFO:teuthology.orchestra.run.smithi069.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-12T15:17:05.976 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-12T15:17:05.977 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-12T15:17:05.977 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-12T15:17:05.977 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-12T15:17:05.977 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:17:05.977 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:17:06.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base 2023-07-12T15:17:06.467 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:17:06.897 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-2:18.1.2-55.gb1bd0540.el8.x86_64 28/32 2023-07-12T15:17:07.123 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-osd-2:18.1.2-55.gb1bd0540.el8.x86_64 29/32 2023-07-12T15:17:07.257 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:34 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:17:07.579 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-osd-2:18.1.2-55.gb1bd0540.el8.x86_64 29/32 2023-07-12T15:17:07.764 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mon-2:18.1.2-55.gb1bd0540.el8.x86_64 30/32 2023-07-12T15:17:07.986 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mon-2:18.1.2-55.gb1bd0540.el8.x86_64 30/32 2023-07-12T15:17:08.163 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mds-2:18.1.2-55.gb1bd0540.el8.x86_64 31/32 2023-07-12T15:17:08.181 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mds-2:18.1.2-55.gb1bd0540.el8.x86_64 31/32 2023-07-12T15:17:08.303 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-2:18.1.2-55.gb1bd0540.el8.x86_64 32/32 2023-07-12T15:17:08.580 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-base-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:17:08.646 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-2:18.1.2-55.gb1bd0540.el8.x86_64 32/32 2023-07-12T15:17:08.647 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-2:18.1.2-55.gb1bd0540.el8.x86_64 1/32 2023-07-12T15:17:08.647 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mds-2:18.1.2-55.gb1bd0540.el8.x86_64 2/32 2023-07-12T15:17:08.647 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-2:18.1.2-55.gb1bd0540.el8.x86_64 3/32 2023-07-12T15:17:08.648 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mon-2:18.1.2-55.gb1bd0540.el8.x86_64 4/32 2023-07-12T15:17:08.648 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-osd-2:18.1.2-55.gb1bd0540.el8.x86_64 5/32 2023-07-12T15:17:08.648 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-modules-core-2:18.1.2-55.gb1bd0540.el8.no 6/32 2023-07-12T15:17:08.649 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2023-07-12T15:17:08.649 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2023-07-12T15:17:08.649 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2023-07-12T15:17:08.650 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2023-07-12T15:17:08.650 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2023-07-12T15:17:08.650 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2023-07-12T15:17:08.650 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2023-07-12T15:17:08.651 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2023-07-12T15:17:08.651 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2023-07-12T15:17:08.651 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2023-07-12T15:17:08.652 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2023-07-12T15:17:08.652 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2023-07-12T15:17:08.652 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2023-07-12T15:17:08.653 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2023-07-12T15:17:08.653 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2023-07-12T15:17:08.653 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2023-07-12T15:17:08.653 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2023-07-12T15:17:08.654 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2023-07-12T15:17:08.654 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 25/32 2023-07-12T15:17:08.654 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/32 2023-07-12T15:17:08.655 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/32 2023-07-12T15:17:08.655 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2023-07-12T15:17:08.655 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 29/32 2023-07-12T15:17:08.655 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 30/32 2023-07-12T15:17:08.656 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 31/32 2023-07-12T15:17:08.689 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:17:08.691 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-07-12T15:17:08.691 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:17:09.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install cephadm 2023-07-12T15:17:09.378 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:17:10.200 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 32/32 2023-07-12T15:17:10.201 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:36 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:17:10.217 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:37 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:17:11.603 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:17:11.603 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:17:11.604 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-07-12T15:17:11.604 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:17:11.604 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:17:11.604 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 160 k 2023-07-12T15:17:11.604 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:17:11.604 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:17:11.605 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:17:11.605 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-07-12T15:17:11.605 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:17:11.605 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 160 k 2023-07-12T15:17:11.605 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 639 k 2023-07-12T15:17:11.605 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:17:12.147 INFO:teuthology.orchestra.run.smithi069.stdout:cephadm-18.1.2-55.gb1bd0540.el8.noarch.rpm 295 kB/s | 160 kB 00:00 2023-07-12T15:17:12.148 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:17:12.148 INFO:teuthology.orchestra.run.smithi069.stdout:Total 294 kB/s | 160 kB 00:00 2023-07-12T15:17:12.148 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:17:12.156 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:17:12.157 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:17:12.159 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:17:12.163 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:17:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:17:12.976 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:17:13.167 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:13.168 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:17:13.168 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:17:13.168 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:17:13.168 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:17:13.169 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:17:13.169 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:17:13.169 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:17:13.169 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-12T15:17:13.169 INFO:teuthology.orchestra.run.smithi145.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-12T15:17:13.169 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-12T15:17:13.169 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-12T15:17:13.170 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-12T15:17:13.170 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-12T15:17:13.170 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-12T15:17:13.170 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-12T15:17:13.170 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-12T15:17:13.170 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-12T15:17:13.171 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-12T15:17:13.171 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-12T15:17:13.171 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-12T15:17:13.171 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend-2.6-1.el8.noarch 2023-07-12T15:17:13.171 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-12T15:17:13.171 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-12T15:17:13.171 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-12T15:17:13.172 INFO:teuthology.orchestra.run.smithi145.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-12T15:17:13.172 INFO:teuthology.orchestra.run.smithi145.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-12T15:17:13.172 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-12T15:17:13.172 INFO:teuthology.orchestra.run.smithi145.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-12T15:17:13.172 INFO:teuthology.orchestra.run.smithi145.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-12T15:17:13.172 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-12T15:17:13.172 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-12T15:17:13.173 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-12T15:17:13.173 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-12T15:17:13.173 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:13.173 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:17:13.371 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-base 2023-07-12T15:17:13.719 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:17:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:17:14.055 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:17:14.222 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:17:14.524 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:40 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:17:15.248 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:17:15.249 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:42 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:17:15.787 INFO:teuthology.orchestra.run.smithi145.stdout:Package ceph-base-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:17:15.896 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:17:15.897 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2023-07-12T15:17:15.898 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:17:16.201 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install cephadm 2023-07-12T15:17:16.549 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:17:17.403 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:43 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:17:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:17:17.821 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:17:17.821 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:17:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:17:17.822 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:17:17.822 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:17:17.960 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-immutable-object-cache 2023-07-12T15:17:18.316 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:17:18.795 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:17:18.796 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:17:18.796 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2023-07-12T15:17:18.797 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:17:18.797 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:17:18.797 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 160 k 2023-07-12T15:17:18.797 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:18.797 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:17:18.797 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:17:18.797 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2023-07-12T15:17:18.798 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:18.798 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 160 k 2023-07-12T15:17:18.798 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 639 k 2023-07-12T15:17:18.798 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:17:18.979 INFO:teuthology.orchestra.run.smithi145.stdout:cephadm-18.1.2-55.gb1bd0540.el8.noarch.rpm 882 kB/s | 160 kB 00:00 2023-07-12T15:17:18.979 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:17:18.979 INFO:teuthology.orchestra.run.smithi145.stdout:Total 877 kB/s | 160 kB 00:00 2023-07-12T15:17:18.980 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:17:18.987 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:17:18.988 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:17:18.989 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:17:18.993 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:17:19.039 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:17:19.161 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:46 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:17:20.551 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:17:20.552 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:17:20.552 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-07-12T15:17:20.552 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:17:20.552 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:17:20.552 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 165 k 2023-07-12T15:17:20.553 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:17:20.553 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:17:20.553 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:17:20.553 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-07-12T15:17:20.553 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:17:20.554 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 165 k 2023-07-12T15:17:20.554 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 503 k 2023-07-12T15:17:20.554 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:17:20.633 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:17:20.820 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:17:20.893 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-immutable-object-cache-18.1.2-55.gb1bd0540 487 kB/s | 165 kB 00:00 2023-07-12T15:17:20.894 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:17:20.894 INFO:teuthology.orchestra.run.smithi069.stdout:Total 484 kB/s | 165 kB 00:00 2023-07-12T15:17:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:17:20.906 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:17:20.906 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:17:20.959 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:17:20.962 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:17:20.989 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:17:21.080 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:17:21.250 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-2:18.1.2-55.gb1bd0540.el 1/1 2023-07-12T15:17:21.604 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.1.2-55.gb1bd0540.el 1/1 2023-07-12T15:17:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:17:22.032 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:48 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:17:22.604 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-2:18.1.2-55.gb1bd0540.el 1/1 2023-07-12T15:17:22.604 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:49 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:17:24.376 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:17:24.560 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:24.561 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:17:24.561 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:17:24.561 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:24.561 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:17:24.697 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-immutable-object-cache 2023-07-12T15:17:25.011 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:17:25.056 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:17:25.197 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:17:25.197 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:17:25.198 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:17:25.198 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:17:25.198 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:17:25.326 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr 2023-07-12T15:17:25.671 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:17:25.841 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:51 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:17:26.504 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:53 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:17:27.220 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:17:27.220 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:17:27.221 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2023-07-12T15:17:27.221 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:17:27.221 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:17:27.221 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 165 k 2023-07-12T15:17:27.221 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:27.221 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:17:27.221 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:17:27.222 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2023-07-12T15:17:27.222 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:27.222 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 165 k 2023-07-12T15:17:27.222 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 503 k 2023-07-12T15:17:27.222 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:17:27.408 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-immutable-object-cache-18.1.2-55.gb1bd0540 885 kB/s | 165 kB 00:00 2023-07-12T15:17:27.409 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:17:27.409 INFO:teuthology.orchestra.run.smithi145.stdout:Total 879 kB/s | 165 kB 00:00 2023-07-12T15:17:27.409 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:17:27.420 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:17:27.421 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:17:27.472 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:17:27.476 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:17:27.600 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:17:27.752 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-immutable-object-cache-2:18.1.2-55.gb1bd0540.el 1/1 2023-07-12T15:17:27.777 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-mgr-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:17:27.887 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:17:27.888 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-07-12T15:17:27.888 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:17:27.998 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-dashboard 2023-07-12T15:17:28.110 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.1.2-55.gb1bd0540.el 1/1 2023-07-12T15:17:28.340 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:17:29.130 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-immutable-object-cache-2:18.1.2-55.gb1bd0540.el 1/1 2023-07-12T15:17:29.131 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:55 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:17:29.177 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:56 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:17:30.569 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:17:30.570 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================================= 2023-07-12T15:17:30.570 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-07-12T15:17:30.570 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================================= 2023-07-12T15:17:30.570 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:17:30.570 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 1.7 M 2023-07-12T15:17:30.571 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-07-12T15:17:30.571 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 23 k 2023-07-12T15:17:30.571 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 15 k 2023-07-12T15:17:30.571 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-12T15:17:30.571 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-12T15:17:30.571 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-12T15:17:30.571 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:17:30.572 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:17:30.572 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================================= 2023-07-12T15:17:30.572 INFO:teuthology.orchestra.run.smithi069.stdout:Install 6 Packages 2023-07-12T15:17:30.572 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:17:30.573 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 2.0 M 2023-07-12T15:17:30.573 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 8.0 M 2023-07-12T15:17:30.573 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:17:30.741 INFO:teuthology.orchestra.run.smithi069.stdout:(1/6): ceph-grafana-dashboards-18.1.2-55.gb1bd0 163 kB/s | 23 kB 00:00 2023-07-12T15:17:30.841 INFO:teuthology.orchestra.run.smithi069.stdout:(2/6): ceph-prometheus-alerts-18.1.2-55.gb1bd05 60 kB/s | 15 kB 00:00 2023-07-12T15:17:31.067 INFO:teuthology.orchestra.run.smithi069.stdout:(3/6): ceph-mgr-dashboard-18.1.2-55.gb1bd0540.e 3.6 MB/s | 1.7 MB 00:00 2023-07-12T15:17:31.142 INFO:teuthology.orchestra.run.smithi069.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 569 kB/s | 43 kB 00:00 2023-07-12T15:17:31.653 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:17:31.837 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:31.838 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:17:31.838 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:17:31.838 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:31.838 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:17:31.969 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr 2023-07-12T15:17:32.318 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:17:33.235 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:59 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:17:34.498 INFO:teuthology.orchestra.run.smithi145.stdout:Package ceph-mgr-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:17:34.608 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:17:34.609 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2023-07-12T15:17:34.609 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:17:34.723 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-dashboard 2023-07-12T15:17:35.068 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:17:35.969 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:01 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:17:37.354 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:17:37.355 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================================= 2023-07-12T15:17:37.355 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2023-07-12T15:17:37.355 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================================= 2023-07-12T15:17:37.355 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:17:37.356 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 1.7 M 2023-07-12T15:17:37.356 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2023-07-12T15:17:37.356 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 23 k 2023-07-12T15:17:37.356 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-prometheus-alerts noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 15 k 2023-07-12T15:17:37.356 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-12T15:17:37.356 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-12T15:17:37.356 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-12T15:17:37.357 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:37.357 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:17:37.357 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================================= 2023-07-12T15:17:37.357 INFO:teuthology.orchestra.run.smithi145.stdout:Install 6 Packages 2023-07-12T15:17:37.357 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:37.358 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 2.0 M 2023-07-12T15:17:37.359 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 8.0 M 2023-07-12T15:17:37.359 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:17:37.538 INFO:teuthology.orchestra.run.smithi145.stdout:(1/6): ceph-prometheus-alerts-18.1.2-55.gb1bd05 95 kB/s | 15 kB 00:00 2023-07-12T15:17:37.680 INFO:teuthology.orchestra.run.smithi145.stdout:(2/6): ceph-mgr-dashboard-18.1.2-55.gb1bd0540.e 5.8 MB/s | 1.7 MB 00:00 2023-07-12T15:17:37.697 INFO:teuthology.orchestra.run.smithi145.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 213 kB/s | 34 kB 00:00 2023-07-12T15:17:37.714 INFO:teuthology.orchestra.run.smithi145.stdout:(4/6): ceph-grafana-dashboards-18.1.2-55.gb1bd0 70 kB/s | 23 kB 00:00 2023-07-12T15:17:37.748 INFO:teuthology.orchestra.run.smithi145.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 846 kB/s | 43 kB 00:00 2023-07-12T15:17:38.716 INFO:teuthology.orchestra.run.smithi145.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 190 kB/s | 196 kB 00:01 2023-07-12T15:17:38.720 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:17:38.721 INFO:teuthology.orchestra.run.smithi145.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2023-07-12T15:17:38.733 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:17:38.743 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:17:38.744 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:17:38.825 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:17:38.829 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:17:39.207 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:17:39.392 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-07-12T15:17:39.631 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-07-12T15:17:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-07-12T15:17:39.981 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-prometheus-alerts-2:18.1.2-55.gb1bd0540.el8.noa 4/6 2023-07-12T15:17:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-grafana-dashboards-2:18.1.2-55.gb1bd0540.el8.no 5/6 2023-07-12T15:17:40.572 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-dashboard-2:18.1.2-55.gb1bd0540.el8.noarch 6/6 2023-07-12T15:17:40.805 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.1.2-55.gb1bd0540.el8.noarch 6/6 2023-07-12T15:17:40.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-grafana-dashboards-2:18.1.2-55.gb1bd0540.el8.no 1/6 2023-07-12T15:17:40.806 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-dashboard-2:18.1.2-55.gb1bd0540.el8.noarch 2/6 2023-07-12T15:17:40.806 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-prometheus-alerts-2:18.1.2-55.gb1bd0540.el8.noa 3/6 2023-07-12T15:17:40.806 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-07-12T15:17:40.806 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-07-12T15:17:41.881 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-07-12T15:17:41.881 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:07 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:17:44.199 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:17:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:44.387 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:17:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:17:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:17:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-prometheus-alerts-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:17:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-12T15:17:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-12T15:17:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-12T15:17:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:44.388 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:17:44.529 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-diskprediction-local 2023-07-12T15:17:44.875 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:17:45.674 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:11 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:17:47.052 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:17:47.053 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================================================= 2023-07-12T15:17:47.053 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2023-07-12T15:17:47.053 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================================================= 2023-07-12T15:17:47.053 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:17:47.054 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 7.4 M 2023-07-12T15:17:47.054 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2023-07-12T15:17:47.054 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-12T15:17:47.054 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-12T15:17:47.054 INFO:teuthology.orchestra.run.smithi145.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-12T15:17:47.054 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-12T15:17:47.054 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-12T15:17:47.055 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-12T15:17:47.055 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-12T15:17:47.055 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:47.055 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:17:47.055 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================================================= 2023-07-12T15:17:47.055 INFO:teuthology.orchestra.run.smithi145.stdout:Install 8 Packages 2023-07-12T15:17:47.055 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:17:47.056 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 35 M 2023-07-12T15:17:47.056 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 210 M 2023-07-12T15:17:47.056 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:17:47.188 INFO:teuthology.orchestra.run.smithi145.stdout:(1/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2023-07-12T15:17:47.230 INFO:teuthology.orchestra.run.smithi145.stdout:(2/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 5.3 MB/s | 225 kB 00:00 2023-07-12T15:17:47.380 INFO:teuthology.orchestra.run.smithi145.stdout:(3/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 32 MB/s | 4.9 MB 00:00 2023-07-12T15:17:47.547 INFO:teuthology.orchestra.run.smithi145.stdout:(4/8): python3-scipy-1.0.0-21.module+el8.5.0+10 28 MB/s | 14 MB 00:00 2023-07-12T15:17:47.631 INFO:teuthology.orchestra.run.smithi145.stdout:(5/8): libgfortran-8.5.0-10.1.el8_6.x86_64.rpm 7.6 MB/s | 644 kB 00:00 2023-07-12T15:17:47.706 INFO:teuthology.orchestra.run.smithi145.stdout:(6/8): openblas-0.3.15-3.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-07-12T15:17:47.731 INFO:teuthology.orchestra.run.smithi145.stdout:(7/8): libquadmath-8.5.0-10.1.el8_6.x86_64.rpm 1.7 MB/s | 171 kB 00:00 2023-07-12T15:17:47.823 INFO:teuthology.orchestra.run.smithi145.stdout:(8/8): ceph-mgr-diskprediction-local-18.1.2-55. 9.6 MB/s | 7.4 MB 00:00 2023-07-12T15:17:47.824 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:17:47.824 INFO:teuthology.orchestra.run.smithi145.stdout:Total 46 MB/s | 35 MB 00:00 2023-07-12T15:17:48.061 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:17:48.080 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:17:48.080 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:17:48.372 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:17:48.376 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:17:48.965 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:17:49.114 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-07-12T15:17:49.258 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-07-12T15:17:49.431 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-07-12T15:17:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-07-12T15:17:50.197 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-07-12T15:17:50.796 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-07-12T15:17:51.009 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-0.3.15-3.el8.x86_64 4/8 2023-07-12T15:17:51.696 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 4/8 2023-07-12T15:17:51.983 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-07-12T15:17:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-07-12T15:17:53.978 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-07-12T15:17:55.659 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 7/8 2023-07-12T15:17:55.821 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-diskprediction-local-2:18.1.2-55.gb1bd0540. 8/8 2023-07-12T15:17:56.087 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.1.2-55.gb1bd0540. 8/8 2023-07-12T15:17:56.087 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.1.2-55.gb1bd0540. 1/8 2023-07-12T15:17:56.087 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 2/8 2023-07-12T15:17:56.087 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 3/8 2023-07-12T15:17:56.088 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 4/8 2023-07-12T15:17:56.088 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 5/8 2023-07-12T15:17:56.088 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 6/8 2023-07-12T15:17:56.088 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 7/8 2023-07-12T15:17:57.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 8/8 2023-07-12T15:17:57.169 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:23 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:18:00.186 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:18:00.372 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:18:00.373 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:18:00.373 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-12T15:18:00.373 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-12T15:18:00.373 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-12T15:18:00.374 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-12T15:18:00.374 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-12T15:18:00.374 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-12T15:18:00.374 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-12T15:18:00.374 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:18:00.511 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-rook 2023-07-12T15:18:00.858 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:18:02.130 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:28 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:18:03.589 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:18:03.591 INFO:teuthology.orchestra.run.smithi145.stdout:====================================================================================================== 2023-07-12T15:18:03.591 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2023-07-12T15:18:03.591 INFO:teuthology.orchestra.run.smithi145.stdout:====================================================================================================== 2023-07-12T15:18:03.591 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:18:03.591 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 47 k 2023-07-12T15:18:03.591 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2023-07-12T15:18:03.592 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-12T15:18:03.592 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-12T15:18:03.592 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-12T15:18:03.592 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-12T15:18:03.592 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-12T15:18:03.592 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-12T15:18:03.592 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-12T15:18:03.593 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-12T15:18:03.593 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-12T15:18:03.593 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-12T15:18:03.593 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-12T15:18:03.593 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-12T15:18:03.593 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:03.593 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:18:03.594 INFO:teuthology.orchestra.run.smithi145.stdout:====================================================================================================== 2023-07-12T15:18:03.594 INFO:teuthology.orchestra.run.smithi145.stdout:Install 13 Packages 2023-07-12T15:18:03.594 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:03.594 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 1.9 M 2023-07-12T15:18:03.595 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 25 M 2023-07-12T15:18:03.595 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:18:04.237 INFO:teuthology.orchestra.run.smithi145.stdout:(1/13): ceph-mgr-rook-18.1.2-55.gb1bd0540.el8.n 104 kB/s | 47 kB 00:00 2023-07-12T15:18:04.370 INFO:teuthology.orchestra.run.smithi145.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 28 kB/s | 16 kB 00:00 2023-07-12T15:18:04.395 INFO:teuthology.orchestra.run.smithi145.stdout:(3/13): python3-cachetools-3.1.1-4.el8.noarch.r 54 kB/s | 33 kB 00:00 2023-07-12T15:18:04.604 INFO:teuthology.orchestra.run.smithi145.stdout:(4/13): python3-rsa-4.8-1.el8.noarch.rpm 295 kB/s | 61 kB 00:00 2023-07-12T15:18:04.721 INFO:teuthology.orchestra.run.smithi145.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 525 kB/s | 61 kB 00:00 2023-07-12T15:18:04.788 INFO:teuthology.orchestra.run.smithi145.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.6 MB/s | 110 kB 00:00 2023-07-12T15:18:04.813 INFO:teuthology.orchestra.run.smithi145.stdout:(7/13): python3-google-auth-1.1.1-10.el8.noarch 141 kB/s | 81 kB 00:00 2023-07-12T15:18:04.838 INFO:teuthology.orchestra.run.smithi145.stdout:(8/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 529 kB/s | 27 kB 00:00 2023-07-12T15:18:04.897 INFO:teuthology.orchestra.run.smithi145.stdout:(9/13): python3-jsonpointer-1.10-11.el8.noarch. 377 kB/s | 20 kB 00:00 2023-07-12T15:18:04.960 INFO:teuthology.orchestra.run.smithi145.stdout:(10/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 861 kB/s | 126 kB 00:00 2023-07-12T15:18:05.023 INFO:teuthology.orchestra.run.smithi145.stdout:(11/13): python3-requests-oauthlib-1.0.0-1.el8. 341 kB/s | 43 kB 00:00 2023-07-12T15:18:05.048 INFO:teuthology.orchestra.run.smithi145.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 1.7 MB/s | 155 kB 00:00 2023-07-12T15:18:06.198 INFO:teuthology.orchestra.run.smithi145.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 653 kB/s | 1.2 MB 00:01 2023-07-12T15:18:06.201 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:18:06.201 INFO:teuthology.orchestra.run.smithi145.stdout:Total 757 kB/s | 1.9 MB 00:02 2023-07-12T15:18:06.269 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:18:06.285 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:18:06.286 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:18:06.358 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:18:06.363 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:18:06.900 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:18:07.127 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-07-12T15:18:07.362 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rsa-4.8-1.el8.noarch 2/13 2023-07-12T15:18:07.612 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-07-12T15:18:07.873 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2023-07-12T15:18:08.067 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2023-07-12T15:18:08.234 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2023-07-12T15:18:08.388 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2023-07-12T15:18:08.635 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-07-12T15:18:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-07-12T15:18:09.008 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-07-12T15:18:10.110 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-07-12T15:18:10.694 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-07-12T15:18:10.893 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-rook-2:18.1.2-55.gb1bd0540.el8.noarch 13/13 2023-07-12T15:18:11.123 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-rook-2:18.1.2-55.gb1bd0540.el8.noarch 13/13 2023-07-12T15:18:11.124 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-rook-2:18.1.2-55.gb1bd0540.el8.noarch 1/13 2023-07-12T15:18:11.124 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-07-12T15:18:11.124 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-07-12T15:18:11.124 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-07-12T15:18:11.124 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-07-12T15:18:11.124 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 6/13 2023-07-12T15:18:11.125 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-07-12T15:18:11.125 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 8/13 2023-07-12T15:18:11.125 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 9/13 2023-07-12T15:18:11.125 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 10/13 2023-07-12T15:18:11.125 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-07-12T15:18:11.125 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 12/13 2023-07-12T15:18:12.252 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 13/13 2023-07-12T15:18:12.252 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:38 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:18:14.727 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:18:14.912 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:14.913 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:18:14.913 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:18:14.913 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-12T15:18:14.913 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-12T15:18:14.914 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-12T15:18:14.914 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-12T15:18:14.914 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-12T15:18:14.914 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-12T15:18:14.914 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-12T15:18:14.914 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-12T15:18:14.914 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-12T15:18:14.914 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-12T15:18:14.915 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-12T15:18:14.915 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-12T15:18:14.915 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:14.915 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:18:15.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-cephadm 2023-07-12T15:18:15.417 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:18:16.299 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:42 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:18:17.676 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:18:17.676 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================= 2023-07-12T15:18:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2023-07-12T15:18:17.677 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================= 2023-07-12T15:18:17.677 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:18:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 127 k 2023-07-12T15:18:17.677 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2023-07-12T15:18:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-12T15:18:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-12T15:18:17.678 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-12T15:18:17.678 INFO:teuthology.orchestra.run.smithi145.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-12T15:18:17.678 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:17.678 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:18:17.678 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================= 2023-07-12T15:18:17.678 INFO:teuthology.orchestra.run.smithi145.stdout:Install 5 Packages 2023-07-12T15:18:17.678 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:17.679 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 6.0 M 2023-07-12T15:18:17.679 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 26 M 2023-07-12T15:18:17.679 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:18:18.065 INFO:teuthology.orchestra.run.smithi145.stdout:(1/5): ceph-mgr-cephadm-18.1.2-55.gb1bd0540.el8 356 kB/s | 127 kB 00:00 2023-07-12T15:18:18.123 INFO:teuthology.orchestra.run.smithi145.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 9.0 MB/s | 538 kB 00:00 2023-07-12T15:18:18.165 INFO:teuthology.orchestra.run.smithi145.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 131 kB/s | 60 kB 00:00 2023-07-12T15:18:18.289 INFO:teuthology.orchestra.run.smithi145.stdout:(4/5): python3-babel-2.5.1-7.el8.noarch.rpm 29 MB/s | 4.8 MB 00:00 2023-07-12T15:18:18.449 INFO:teuthology.orchestra.run.smithi145.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 641 kB/s | 476 kB 00:00 2023-07-12T15:18:18.451 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:18:18.451 INFO:teuthology.orchestra.run.smithi145.stdout:Total 7.7 MB/s | 6.0 MB 00:00 2023-07-12T15:18:18.504 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:18:18.515 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:18:18.516 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:18:18.594 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:18:18.598 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:18:19.568 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:18:19.937 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2023-07-12T15:18:20.252 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2023-07-12T15:18:21.234 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-07-12T15:18:21.556 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-07-12T15:18:21.750 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 5/5 2023-07-12T15:18:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 5/5 2023-07-12T15:18:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 1/5 2023-07-12T15:18:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-07-12T15:18:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-07-12T15:18:21.990 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-07-12T15:18:23.025 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 5/5 2023-07-12T15:18:23.025 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:49 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:18:25.342 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:18:25.530 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:25.531 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:18:25.531 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:18:25.532 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-12T15:18:25.532 INFO:teuthology.orchestra.run.smithi145.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-12T15:18:25.532 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-12T15:18:25.532 INFO:teuthology.orchestra.run.smithi145.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-12T15:18:25.532 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:25.533 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:18:25.671 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-fuse 2023-07-12T15:18:26.027 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:18:26.845 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:52 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:18:28.229 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:18:28.229 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:18:28.229 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2023-07-12T15:18:28.230 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:18:28.230 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:18:28.230 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 847 k 2023-07-12T15:18:28.230 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2023-07-12T15:18:28.230 INFO:teuthology.orchestra.run.smithi145.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-12T15:18:28.230 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:28.230 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:18:28.231 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:18:28.231 INFO:teuthology.orchestra.run.smithi145.stdout:Install 2 Packages 2023-07-12T15:18:28.231 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:28.231 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 929 k 2023-07-12T15:18:28.232 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 3.0 M 2023-07-12T15:18:28.232 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:18:28.302 INFO:teuthology.orchestra.run.smithi145.stdout:(1/2): fuse-2.9.7-15.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2023-07-12T15:18:28.627 INFO:teuthology.orchestra.run.smithi145.stdout:(2/2): ceph-fuse-18.1.2-55.gb1bd0540.el8.x86_64 2.1 MB/s | 847 kB 00:00 2023-07-12T15:18:28.628 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:18:28.628 INFO:teuthology.orchestra.run.smithi145.stdout:Total 2.3 MB/s | 929 kB 00:00 2023-07-12T15:18:28.639 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:18:28.660 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:18:28.661 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:18:28.740 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:18:28.740 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:18:28.920 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:18:29.172 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : fuse-2.9.7-15.el8.x86_64 1/2 2023-07-12T15:18:29.305 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 2/2 2023-07-12T15:18:29.629 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 2/2 2023-07-12T15:18:29.629 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 1/2 2023-07-12T15:18:30.699 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 2/2 2023-07-12T15:18:30.699 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:56 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:18:33.062 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:18:33.246 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:33.247 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:18:33.247 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 fuse-2.9.7-15.el8.x86_64 2023-07-12T15:18:33.247 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:33.247 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:18:33.378 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-volume 2023-07-12T15:18:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:18:34.579 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:00 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:18:36.034 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:18:36.035 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:18:36.035 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2023-07-12T15:18:36.035 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:18:36.035 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:18:36.035 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-volume noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 258 k 2023-07-12T15:18:36.036 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:36.036 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:18:36.036 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:18:36.036 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2023-07-12T15:18:36.036 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:36.036 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 258 k 2023-07-12T15:18:36.037 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 1.2 M 2023-07-12T15:18:36.037 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:18:36.364 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-volume-18.1.2-55.gb1bd0540.el8.noarch.rpm 788 kB/s | 258 kB 00:00 2023-07-12T15:18:36.365 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:18:36.365 INFO:teuthology.orchestra.run.smithi145.stdout:Total 785 kB/s | 258 kB 00:00 2023-07-12T15:18:36.365 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:18:36.375 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:18:36.376 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:18:36.393 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:18:36.396 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:18:36.559 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:18:36.847 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-volume-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:18:37.185 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-volume-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:18:38.294 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-volume-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:18:38.295 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:04 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:18:41.052 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:18:41.237 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:41.237 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:18:41.237 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-volume-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:18:41.238 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:41.238 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:18:41.373 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install librados-devel 2023-07-12T15:18:41.731 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:18:42.586 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:08 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:18:43.975 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:18:43.976 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:18:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2023-07-12T15:18:43.977 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:18:43.977 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:18:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 139 k 2023-07-12T15:18:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:43.977 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:18:43.978 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:18:43.978 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2023-07-12T15:18:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:43.978 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 139 k 2023-07-12T15:18:43.979 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 513 k 2023-07-12T15:18:43.979 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:18:44.163 INFO:teuthology.orchestra.run.smithi145.stdout:librados-devel-18.1.2-55.gb1bd0540.el8.x86_64.r 747 kB/s | 139 kB 00:00 2023-07-12T15:18:44.164 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:18:44.165 INFO:teuthology.orchestra.run.smithi145.stdout:Total 742 kB/s | 139 kB 00:00 2023-07-12T15:18:44.165 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:18:44.170 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:18:44.170 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:18:44.223 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:18:44.227 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:18:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:18:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librados-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:18:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:18:45.789 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:18:45.790 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:11 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:18:49.389 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:18:49.573 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:49.574 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:18:49.574 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:18:49.574 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:49.574 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:18:49.708 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install libcephfs2 2023-07-12T15:18:50.064 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:18:51.552 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:17 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:18:52.859 INFO:teuthology.orchestra.run.smithi145.stdout:Package libcephfs2-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:18:52.968 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:18:52.969 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2023-07-12T15:18:52.970 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:18:53.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install libcephfs-devel 2023-07-12T15:18:53.448 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:18:55.000 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:20 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:18:56.407 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:18:56.408 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:18:56.409 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2023-07-12T15:18:56.409 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:18:56.409 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:18:56.409 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 30 k 2023-07-12T15:18:56.410 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:56.410 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:18:56.410 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:18:56.410 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2023-07-12T15:18:56.411 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:18:56.411 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 30 k 2023-07-12T15:18:56.411 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 137 k 2023-07-12T15:18:56.411 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:18:56.634 INFO:teuthology.orchestra.run.smithi145.stdout:libcephfs-devel-18.1.2-55.gb1bd0540.el8.x86_64. 135 kB/s | 30 kB 00:00 2023-07-12T15:18:56.635 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:18:56.635 INFO:teuthology.orchestra.run.smithi145.stdout:Total 134 kB/s | 30 kB 00:00 2023-07-12T15:18:56.635 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:18:56.638 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:18:56.639 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:18:56.649 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:18:56.653 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:18:56.738 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:18:56.899 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:18:57.111 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephfs-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:18:58.110 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:18:58.110 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:24 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:19:02.056 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:19:02.241 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:02.242 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:19:02.242 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:19:02.242 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:02.242 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:19:02.378 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install librados2 2023-07-12T15:19:02.732 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:19:04.227 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:30 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:19:05.497 INFO:teuthology.orchestra.run.smithi145.stdout:Package librados2-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:19:05.607 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:19:05.608 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2023-07-12T15:19:05.613 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:19:05.727 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install librbd1 2023-07-12T15:19:06.082 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:19:06.907 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:32 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:19:08.187 INFO:teuthology.orchestra.run.smithi145.stdout:Package librbd1-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:19:08.296 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:19:08.298 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2023-07-12T15:19:08.298 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:19:08.418 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rados 2023-07-12T15:19:08.773 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:19:09.589 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:35 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:19:10.938 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rados-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:19:11.047 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:19:11.048 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2023-07-12T15:19:11.048 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:19:11.165 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rgw 2023-07-12T15:19:11.519 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:19:12.379 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:38 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:19:13.642 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rgw-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:19:13.752 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:19:13.753 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2023-07-12T15:19:13.753 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:19:13.874 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-cephfs 2023-07-12T15:19:14.230 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:19:15.096 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:41 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:19:16.369 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-cephfs-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:19:16.480 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:19:16.481 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2023-07-12T15:19:16.482 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:19:16.602 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rbd 2023-07-12T15:19:16.956 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:19:17.852 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:43 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:19:19.188 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rbd-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:19:19.302 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:19:19.303 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2023-07-12T15:19:19.303 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:19:19.438 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-fuse 2023-07-12T15:19:19.792 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:19:20.631 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:46 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:19:22.012 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:19:22.012 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:19:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2023-07-12T15:19:22.012 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:19:22.012 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:19:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 94 k 2023-07-12T15:19:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:22.013 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:19:22.013 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:19:22.013 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2023-07-12T15:19:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:22.013 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 94 k 2023-07-12T15:19:22.013 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 280 k 2023-07-12T15:19:22.014 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:19:22.177 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-fuse-18.1.2-55.gb1bd0540.el8.x86_64.rpm 570 kB/s | 94 kB 00:00 2023-07-12T15:19:22.178 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:19:22.178 INFO:teuthology.orchestra.run.smithi145.stdout:Total 566 kB/s | 94 kB 00:00 2023-07-12T15:19:22.178 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:19:22.189 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:19:22.189 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:19:22.244 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:19:22.248 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:19:22.360 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:19:22.511 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:19:22.739 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:19:23.779 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:19:23.779 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:49 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:19:26.254 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:19:26.445 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:26.446 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:19:26.446 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:19:26.446 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:26.446 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:19:26.574 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-mirror 2023-07-12T15:19:26.930 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:19:27.773 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:53 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:19:29.172 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:19:29.172 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:19:29.173 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2023-07-12T15:19:29.173 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:19:29.173 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:19:29.173 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 3.4 M 2023-07-12T15:19:29.173 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:29.173 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:19:29.174 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:19:29.174 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2023-07-12T15:19:29.174 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:29.174 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 3.4 M 2023-07-12T15:19:29.174 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 16 M 2023-07-12T15:19:29.174 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:19:29.614 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-mirror-18.1.2-55.gb1bd0540.el8.x86_64.rpm 7.8 MB/s | 3.4 MB 00:00 2023-07-12T15:19:29.614 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:19:29.614 INFO:teuthology.orchestra.run.smithi145.stdout:Total 7.8 MB/s | 3.4 MB 00:00 2023-07-12T15:19:29.615 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:19:29.627 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:19:29.627 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:19:29.693 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:19:29.697 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:19:30.136 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:19:30.312 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-mirror-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:19:30.665 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-mirror-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:19:31.720 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-mirror-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:19:31.720 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:03:57 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:19:34.606 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:19:34.793 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:34.793 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:19:34.793 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:19:34.794 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:34.794 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:19:34.946 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-nbd 2023-07-12T15:19:35.305 INFO:teuthology.orchestra.run.smithi145.stdout:Updating Subscription Management repositories. 2023-07-12T15:19:36.205 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:04:02 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:19:37.657 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-07-12T15:19:37.658 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:19:37.659 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2023-07-12T15:19:37.659 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:19:37.659 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-07-12T15:19:37.659 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 179 k 2023-07-12T15:19:37.659 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:37.659 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-07-12T15:19:37.660 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-07-12T15:19:37.660 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2023-07-12T15:19:37.660 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:37.660 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 179 k 2023-07-12T15:19:37.660 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 568 k 2023-07-12T15:19:37.660 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-07-12T15:19:38.000 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-nbd-18.1.2-55.gb1bd0540.el8.x86_64.rpm 527 kB/s | 179 kB 00:00 2023-07-12T15:19:38.001 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:19:38.001 INFO:teuthology.orchestra.run.smithi145.stdout:Total 525 kB/s | 179 kB 00:00 2023-07-12T15:19:38.001 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-07-12T15:19:38.012 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-07-12T15:19:38.013 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-07-12T15:19:38.066 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-07-12T15:19:38.070 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-07-12T15:19:38.201 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-07-12T15:19:38.383 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-nbd-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:19:38.612 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-nbd-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:19:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-nbd-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:19:39.718 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:04:05 ago on Wed 12 Jul 2023 03:15:34 PM UTC. 2023-07-12T15:19:41.838 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] python3-repoze-lru-0.7-6.el8.noarch.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/p/python3-repoze-lru-0.7-6.el8.noarch.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-07-12T15:19:41.839 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] python3-routes-2.4.1-12.el8.noarch.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/p/python3-routes-2.4.1-12.el8.noarch.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-07-12T15:19:42.415 INFO:teuthology.orchestra.run.smithi069.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 261 B/s | 34 kB 02:11 2023-07-12T15:19:42.657 INFO:teuthology.orchestra.run.smithi069.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.5 kB/s | 196 kB 02:11 2023-07-12T15:19:42.658 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:19:42.659 INFO:teuthology.orchestra.run.smithi069.stdout:Total 15 kB/s | 2.0 MB 02:12 2023-07-12T15:19:42.671 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:19:42.681 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:19:42.682 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:19:42.743 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:19:42.747 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:19:42.843 INFO:teuthology.orchestra.run.smithi145.stdout:Installed products updated. 2023-07-12T15:19:43.006 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:19:43.027 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:43.027 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-07-12T15:19:43.028 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:19:43.028 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:19:43.028 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-07-12T15:19:43.189 DEBUG:teuthology.parallel:result is None 2023-07-12T15:19:43.220 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-07-12T15:19:43.425 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-07-12T15:19:43.638 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-07-12T15:19:43.742 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-prometheus-alerts-2:18.1.2-55.gb1bd0540.el8.noa 4/6 2023-07-12T15:19:44.062 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-grafana-dashboards-2:18.1.2-55.gb1bd0540.el8.no 5/6 2023-07-12T15:19:44.265 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-dashboard-2:18.1.2-55.gb1bd0540.el8.noarch 6/6 2023-07-12T15:19:44.495 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.1.2-55.gb1bd0540.el8.noarch 6/6 2023-07-12T15:19:44.495 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-grafana-dashboards-2:18.1.2-55.gb1bd0540.el8.no 1/6 2023-07-12T15:19:44.495 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-dashboard-2:18.1.2-55.gb1bd0540.el8.noarch 2/6 2023-07-12T15:19:44.496 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-prometheus-alerts-2:18.1.2-55.gb1bd0540.el8.noa 3/6 2023-07-12T15:19:44.496 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-07-12T15:19:44.496 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-07-12T15:19:45.537 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-07-12T15:19:45.537 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:12 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:19:50.175 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:19:50.358 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:19:50.359 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:19:50.359 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:19:50.359 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:19:50.359 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:19:50.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-12T15:19:50.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-12T15:19:50.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-12T15:19:50.360 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:19:50.360 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:19:50.493 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-diskprediction-local 2023-07-12T15:19:50.842 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:19:52.282 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:19 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:19:53.656 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:19:53.657 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================================================= 2023-07-12T15:19:53.658 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-07-12T15:19:53.658 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================================================= 2023-07-12T15:19:53.658 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:19:53.659 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 7.4 M 2023-07-12T15:19:53.659 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-07-12T15:19:53.659 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-12T15:19:53.659 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-12T15:19:53.659 INFO:teuthology.orchestra.run.smithi069.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-12T15:19:53.660 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-12T15:19:53.660 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-12T15:19:53.660 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-12T15:19:53.660 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-12T15:19:53.661 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:19:53.661 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:19:53.661 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================================================= 2023-07-12T15:19:53.661 INFO:teuthology.orchestra.run.smithi069.stdout:Install 8 Packages 2023-07-12T15:19:53.661 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:19:53.662 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 35 M 2023-07-12T15:19:53.662 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 210 M 2023-07-12T15:19:53.662 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:19:53.837 INFO:teuthology.orchestra.run.smithi069.stdout:(1/8): libquadmath-8.5.0-10.1.el8_6.x86_64.rpm 969 kB/s | 171 kB 00:00 2023-07-12T15:19:53.887 INFO:teuthology.orchestra.run.smithi069.stdout:(2/8): libgfortran-8.5.0-10.1.el8_6.x86_64.rpm 2.8 MB/s | 644 kB 00:00 2023-07-12T15:19:54.123 INFO:teuthology.orchestra.run.smithi069.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 16 MB/s | 3.7 MB 00:00 2023-07-12T15:19:54.329 INFO:teuthology.orchestra.run.smithi069.stdout:(4/8): python3-scipy-1.0.0-21.module+el8.5.0+10 28 MB/s | 14 MB 00:00 2023-07-12T15:19:54.354 INFO:teuthology.orchestra.run.smithi069.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 964 kB/s | 225 kB 00:00 2023-07-12T15:19:54.488 INFO:teuthology.orchestra.run.smithi069.stdout:(6/8): ceph-mgr-diskprediction-local-18.1.2-55. 8.9 MB/s | 7.4 MB 00:00 2023-07-12T15:19:54.638 INFO:teuthology.orchestra.run.smithi069.stdout:(7/8): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-07-12T15:19:54.705 INFO:teuthology.orchestra.run.smithi069.stdout:(8/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2023-07-12T15:19:54.706 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:19:54.706 INFO:teuthology.orchestra.run.smithi069.stdout:Total 34 MB/s | 35 MB 00:01 2023-07-12T15:19:54.946 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:19:54.964 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:19:54.964 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:19:55.255 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:19:55.259 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:19:55.830 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:19:56.008 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-07-12T15:19:56.160 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-07-12T15:19:56.334 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-07-12T15:19:56.936 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-07-12T15:19:57.134 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-07-12T15:19:57.711 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-07-12T15:19:57.941 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-0.3.15-3.el8.x86_64 4/8 2023-07-12T15:19:58.626 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 4/8 2023-07-12T15:19:58.978 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-07-12T15:19:59.216 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-07-12T15:20:00.967 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-07-12T15:20:02.257 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 7/8 2023-07-12T15:20:02.424 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-diskprediction-local-2:18.1.2-55.gb1bd0540. 8/8 2023-07-12T15:20:02.713 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.1.2-55.gb1bd0540. 8/8 2023-07-12T15:20:02.713 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.1.2-55.gb1bd0540. 1/8 2023-07-12T15:20:02.713 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-07-12T15:20:02.713 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 3/8 2023-07-12T15:20:02.713 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 4/8 2023-07-12T15:20:02.714 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-07-12T15:20:02.714 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-07-12T15:20:02.714 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 7/8 2023-07-12T15:20:03.751 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 8/8 2023-07-12T15:20:03.751 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:30 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:20:08.642 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:20:08.830 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:08.830 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:20:08.830 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:20:08.831 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-12T15:20:08.831 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-12T15:20:08.831 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-12T15:20:08.831 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-12T15:20:08.831 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-12T15:20:08.831 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-12T15:20:08.831 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-12T15:20:08.832 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:08.832 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:20:08.954 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-rook 2023-07-12T15:20:09.304 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:20:10.827 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:37 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:20:12.219 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:20:12.220 INFO:teuthology.orchestra.run.smithi069.stdout:====================================================================================================== 2023-07-12T15:20:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-07-12T15:20:12.221 INFO:teuthology.orchestra.run.smithi069.stdout:====================================================================================================== 2023-07-12T15:20:12.221 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:20:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 47 k 2023-07-12T15:20:12.221 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-07-12T15:20:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-12T15:20:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-12T15:20:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-12T15:20:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-12T15:20:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-12T15:20:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-12T15:20:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-12T15:20:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-12T15:20:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-12T15:20:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-12T15:20:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-12T15:20:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-12T15:20:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:12.223 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:20:12.223 INFO:teuthology.orchestra.run.smithi069.stdout:====================================================================================================== 2023-07-12T15:20:12.223 INFO:teuthology.orchestra.run.smithi069.stdout:Install 13 Packages 2023-07-12T15:20:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:12.224 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 1.9 M 2023-07-12T15:20:12.224 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 25 M 2023-07-12T15:20:12.225 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:20:12.419 INFO:teuthology.orchestra.run.smithi069.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 103 kB/s | 16 kB 00:00 2023-07-12T15:20:12.445 INFO:teuthology.orchestra.run.smithi069.stdout:(2/13): ceph-mgr-rook-18.1.2-55.gb1bd0540.el8.n 257 kB/s | 47 kB 00:00 2023-07-12T15:20:12.978 INFO:teuthology.orchestra.run.smithi069.stdout:(3/13): python3-kubernetes-11.0.0-6.el8.noarch. 2.2 MB/s | 1.2 MB 00:00 2023-07-12T15:20:13.003 INFO:teuthology.orchestra.run.smithi069.stdout:(4/13): python3-google-auth-1.1.1-10.el8.noarch 139 kB/s | 81 kB 00:00 2023-07-12T15:20:13.099 INFO:teuthology.orchestra.run.smithi069.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 641 kB/s | 61 kB 00:00 2023-07-12T15:20:13.129 INFO:teuthology.orchestra.run.smithi069.stdout:(6/13): python3-rsa-4.8-1.el8.noarch.rpm 409 kB/s | 61 kB 00:00 2023-07-12T15:20:13.304 INFO:teuthology.orchestra.run.smithi069.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 208 kB/s | 43 kB 00:00 2023-07-12T15:20:13.355 INFO:teuthology.orchestra.run.smithi069.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 687 kB/s | 155 kB 00:00 2023-07-12T15:20:13.405 INFO:teuthology.orchestra.run.smithi069.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 29 kB/s | 33 kB 00:01 2023-07-12T15:20:13.438 INFO:teuthology.orchestra.run.smithi069.stdout:(10/13): python3-pyasn1-modules-0.3.7-6.el8.noa 822 kB/s | 110 kB 00:00 2023-07-12T15:20:13.522 INFO:teuthology.orchestra.run.smithi069.stdout:(11/13): python3-jsonpatch-1.21-2.el8.noarch.rp 159 kB/s | 27 kB 00:00 2023-07-12T15:20:13.572 INFO:teuthology.orchestra.run.smithi069.stdout:(12/13): python3-jsonpointer-1.10-11.el8.noarch 156 kB/s | 20 kB 00:00 2023-07-12T15:20:13.597 INFO:teuthology.orchestra.run.smithi069.stdout:(13/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 657 kB/s | 126 kB 00:00 2023-07-12T15:20:13.598 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:20:13.599 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2023-07-12T15:20:13.666 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:20:13.682 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:20:13.682 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:20:13.752 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:20:13.756 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:20:14.304 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:20:14.514 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-07-12T15:20:14.732 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rsa-4.8-1.el8.noarch 2/13 2023-07-12T15:20:14.904 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-07-12T15:20:15.055 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-07-12T15:20:15.303 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-07-12T15:20:15.518 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-07-12T15:20:15.759 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-07-12T15:20:16.021 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-07-12T15:20:16.187 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-07-12T15:20:16.445 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-07-12T15:20:17.524 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-07-12T15:20:17.814 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-07-12T15:20:17.989 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-rook-2:18.1.2-55.gb1bd0540.el8.noarch 13/13 2023-07-12T15:20:18.234 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-rook-2:18.1.2-55.gb1bd0540.el8.noarch 13/13 2023-07-12T15:20:18.235 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-rook-2:18.1.2-55.gb1bd0540.el8.noarch 1/13 2023-07-12T15:20:18.235 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-07-12T15:20:18.235 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-07-12T15:20:18.236 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-07-12T15:20:18.236 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-07-12T15:20:18.236 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 6/13 2023-07-12T15:20:18.236 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-07-12T15:20:18.237 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2023-07-12T15:20:18.237 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2023-07-12T15:20:18.237 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-07-12T15:20:18.238 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 11/13 2023-07-12T15:20:18.238 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 12/13 2023-07-12T15:20:19.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-07-12T15:20:19.413 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:46 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:20:24.631 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:20:24.815 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:24.815 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:20:24.815 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:20:24.815 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-12T15:20:24.815 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-12T15:20:24.815 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-12T15:20:24.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-12T15:20:24.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-12T15:20:24.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-12T15:20:24.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-12T15:20:24.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-12T15:20:24.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-12T15:20:24.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-12T15:20:24.817 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-12T15:20:24.817 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-12T15:20:24.817 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:24.817 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:20:24.947 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-cephadm 2023-07-12T15:20:25.305 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:20:26.784 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:53 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:20:28.162 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:20:28.163 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================= 2023-07-12T15:20:28.163 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-07-12T15:20:28.163 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================= 2023-07-12T15:20:28.163 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:20:28.163 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 127 k 2023-07-12T15:20:28.164 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-07-12T15:20:28.164 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-12T15:20:28.164 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-12T15:20:28.164 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-12T15:20:28.164 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-12T15:20:28.164 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:28.165 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:20:28.165 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================= 2023-07-12T15:20:28.165 INFO:teuthology.orchestra.run.smithi069.stdout:Install 5 Packages 2023-07-12T15:20:28.165 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:28.166 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 6.0 M 2023-07-12T15:20:28.166 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 26 M 2023-07-12T15:20:28.166 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:20:28.683 INFO:teuthology.orchestra.run.smithi069.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 178 kB/s | 60 kB 00:00 2023-07-12T15:20:28.932 INFO:teuthology.orchestra.run.smithi069.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 807 kB/s | 476 kB 00:00 2023-07-12T15:20:29.174 INFO:teuthology.orchestra.run.smithi069.stdout:(3/5): ceph-mgr-cephadm-18.1.2-55.gb1bd0540.el8 153 kB/s | 127 kB 00:00 2023-07-12T15:20:29.469 INFO:teuthology.orchestra.run.smithi069.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 682 kB/s | 538 kB 00:00 2023-07-12T15:20:29.575 INFO:teuthology.orchestra.run.smithi069.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 7.4 MB/s | 4.8 MB 00:00 2023-07-12T15:20:29.576 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:20:29.577 INFO:teuthology.orchestra.run.smithi069.stdout:Total 4.2 MB/s | 6.0 MB 00:01 2023-07-12T15:20:29.633 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:20:29.645 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:20:29.645 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:20:29.725 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:20:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:20:30.677 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:20:31.050 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2023-07-12T15:20:31.256 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2023-07-12T15:20:31.576 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-07-12T15:20:31.917 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-07-12T15:20:32.137 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 5/5 2023-07-12T15:20:32.366 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 5/5 2023-07-12T15:20:32.367 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 1/5 2023-07-12T15:20:32.367 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-07-12T15:20:32.367 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-07-12T15:20:32.367 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-07-12T15:20:33.383 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 5/5 2023-07-12T15:20:33.384 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:00 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:20:38.297 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:20:38.481 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:38.481 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:20:38.481 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:20:38.481 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-12T15:20:38.481 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-12T15:20:38.481 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-12T15:20:38.482 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-12T15:20:38.482 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:38.482 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:20:38.607 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-fuse 2023-07-12T15:20:38.966 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:20:40.486 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:07 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:20:41.943 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:20:41.943 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:20:41.943 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-07-12T15:20:41.944 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:20:41.944 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:20:41.944 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 847 k 2023-07-12T15:20:41.944 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2023-07-12T15:20:41.944 INFO:teuthology.orchestra.run.smithi069.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-12T15:20:41.944 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:41.944 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:20:41.945 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:20:41.945 INFO:teuthology.orchestra.run.smithi069.stdout:Install 2 Packages 2023-07-12T15:20:41.945 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:41.945 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 929 k 2023-07-12T15:20:41.945 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 3.0 M 2023-07-12T15:20:41.946 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:20:42.159 INFO:teuthology.orchestra.run.smithi069.stdout:(1/2): fuse-2.9.7-15.el8.x86_64.rpm 386 kB/s | 83 kB 00:00 2023-07-12T15:20:42.551 INFO:teuthology.orchestra.run.smithi069.stdout:(2/2): ceph-fuse-18.1.2-55.gb1bd0540.el8.x86_64 1.4 MB/s | 847 kB 00:00 2023-07-12T15:20:42.551 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:20:42.552 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.5 MB/s | 929 kB 00:00 2023-07-12T15:20:42.563 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:20:42.584 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:20:42.584 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:20:42.658 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:20:42.663 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:20:42.814 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:20:43.092 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : fuse-2.9.7-15.el8.x86_64 1/2 2023-07-12T15:20:43.283 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 2/2 2023-07-12T15:20:43.611 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 2/2 2023-07-12T15:20:43.611 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 1/2 2023-07-12T15:20:44.683 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 2/2 2023-07-12T15:20:44.684 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:11 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:20:48.177 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:20:48.361 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:48.361 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:20:48.361 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 fuse-2.9.7-15.el8.x86_64 2023-07-12T15:20:48.362 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:48.362 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:20:48.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-volume 2023-07-12T15:20:48.848 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:20:49.801 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:16 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:20:51.181 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:20:51.181 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:20:51.182 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2023-07-12T15:20:51.182 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:20:51.182 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:20:51.182 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume noarch 2:18.1.2-55.gb1bd0540.el8 ceph-noarch 258 k 2023-07-12T15:20:51.182 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:51.182 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:20:51.182 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:20:51.183 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-07-12T15:20:51.183 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:51.183 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 258 k 2023-07-12T15:20:51.183 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.2 M 2023-07-12T15:20:51.184 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:20:51.380 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-volume-18.1.2-55.gb1bd0540.el8.noarch.rpm 1.3 MB/s | 258 kB 00:00 2023-07-12T15:20:51.381 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:20:51.381 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.3 MB/s | 258 kB 00:00 2023-07-12T15:20:51.382 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:20:51.392 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:20:51.392 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:20:51.408 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:20:51.412 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:20:51.602 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:20:52.083 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-volume-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:20:52.444 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-volume-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:20:53.471 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-volume-2:18.1.2-55.gb1bd0540.el8.noarch 1/1 2023-07-12T15:20:53.471 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:20 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:20:57.328 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:20:57.513 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:57.513 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:20:57.513 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume-2:18.1.2-55.gb1bd0540.el8.noarch 2023-07-12T15:20:57.513 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:20:57.513 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:20:57.642 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados-devel 2023-07-12T15:20:57.996 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:20:59.333 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:26 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:00.733 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:21:00.734 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:00.734 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-07-12T15:21:00.734 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:00.735 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:21:00.735 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 139 k 2023-07-12T15:21:00.735 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:00.735 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:21:00.735 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:00.735 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-07-12T15:21:00.735 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:00.736 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 139 k 2023-07-12T15:21:00.736 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 513 k 2023-07-12T15:21:00.736 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:21:00.914 INFO:teuthology.orchestra.run.smithi069.stdout:librados-devel-18.1.2-55.gb1bd0540.el8.x86_64.r 778 kB/s | 139 kB 00:00 2023-07-12T15:21:00.915 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:21:00.915 INFO:teuthology.orchestra.run.smithi069.stdout:Total 772 kB/s | 139 kB 00:00 2023-07-12T15:21:00.915 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:21:00.920 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:21:00.921 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:21:00.974 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:21:00.977 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:21:01.090 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:21:01.229 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:21:01.482 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:21:02.489 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:21:02.490 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:29 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:06.392 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:21:06.573 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:06.574 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:21:06.574 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:21:06.574 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:06.574 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:21:06.741 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs2 2023-07-12T15:21:07.097 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:21:08.367 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:35 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:09.659 INFO:teuthology.orchestra.run.smithi069.stdout:Package libcephfs2-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:21:09.767 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:21:09.768 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-07-12T15:21:09.768 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:21:09.887 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs-devel 2023-07-12T15:21:10.242 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:21:11.526 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:38 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:12.905 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:21:12.905 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2023-07-12T15:21:12.906 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:12.906 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:21:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 30 k 2023-07-12T15:21:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:12.906 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:21:12.907 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:12.907 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-07-12T15:21:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:12.907 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 30 k 2023-07-12T15:21:12.907 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 137 k 2023-07-12T15:21:12.907 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:21:13.126 INFO:teuthology.orchestra.run.smithi069.stdout:libcephfs-devel-18.1.2-55.gb1bd0540.el8.x86_64. 139 kB/s | 30 kB 00:00 2023-07-12T15:21:13.127 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:21:13.127 INFO:teuthology.orchestra.run.smithi069.stdout:Total 138 kB/s | 30 kB 00:00 2023-07-12T15:21:13.127 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:21:13.130 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:21:13.131 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:21:13.141 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:21:13.145 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:21:13.224 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:21:13.394 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:21:13.594 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:21:14.558 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:21:14.558 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:41 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:18.067 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:21:18.248 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:18.248 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:21:18.248 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:21:18.248 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:18.248 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:21:18.371 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados2 2023-07-12T15:21:18.725 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:21:20.093 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:47 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:21.359 INFO:teuthology.orchestra.run.smithi069.stdout:Package librados2-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:21:21.468 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:21:21.469 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-07-12T15:21:21.469 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:21:21.583 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librbd1 2023-07-12T15:21:21.935 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:21:23.353 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:50 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:24.702 INFO:teuthology.orchestra.run.smithi069.stdout:Package librbd1-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:21:24.813 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:21:24.814 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-07-12T15:21:24.814 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:21:24.925 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rados 2023-07-12T15:21:25.278 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:21:26.922 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:53 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:28.216 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rados-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:21:28.323 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:21:28.325 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-07-12T15:21:28.325 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:21:28.434 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rgw 2023-07-12T15:21:28.787 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:21:30.406 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:05:57 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:31.675 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rgw-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:21:31.785 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:21:31.786 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-07-12T15:21:31.786 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:21:31.900 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-cephfs 2023-07-12T15:21:32.254 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:21:33.907 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:00 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:35.258 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-cephfs-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:21:35.366 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:21:35.367 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-07-12T15:21:35.367 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:21:35.484 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rbd 2023-07-12T15:21:35.840 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:21:37.252 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:04 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:38.517 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rbd-2:18.1.2-55.gb1bd0540.el8.x86_64 is already installed. 2023-07-12T15:21:38.629 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:21:38.630 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-07-12T15:21:38.630 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:21:38.742 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-fuse 2023-07-12T15:21:39.094 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:21:40.865 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:07 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:42.270 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:21:42.271 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:42.271 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2023-07-12T15:21:42.271 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:42.272 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:21:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 94 k 2023-07-12T15:21:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:42.272 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:21:42.272 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:42.272 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-07-12T15:21:42.272 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:42.273 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 94 k 2023-07-12T15:21:42.273 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 280 k 2023-07-12T15:21:42.273 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:21:42.444 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-fuse-18.1.2-55.gb1bd0540.el8.x86_64.rpm 548 kB/s | 94 kB 00:00 2023-07-12T15:21:42.445 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:21:42.445 INFO:teuthology.orchestra.run.smithi069.stdout:Total 544 kB/s | 94 kB 00:00 2023-07-12T15:21:42.445 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:21:42.455 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:21:42.456 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:21:42.508 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:21:42.512 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:21:42.627 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:21:42.789 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:21:43.005 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:21:44.062 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:21:44.062 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:11 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:47.443 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:21:47.627 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:21:47.628 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:21:47.628 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:21:47.757 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-mirror 2023-07-12T15:21:48.113 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:21:49.001 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:15 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:50.381 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:21:50.382 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:50.382 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2023-07-12T15:21:50.382 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:50.382 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:21:50.382 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 3.4 M 2023-07-12T15:21:50.382 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:50.382 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:21:50.383 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:50.383 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-07-12T15:21:50.383 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:50.383 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 3.4 M 2023-07-12T15:21:50.384 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 16 M 2023-07-12T15:21:50.384 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:21:50.913 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-mirror-18.1.2-55.gb1bd0540.el8.x86_64.rpm 6.5 MB/s | 3.4 MB 00:00 2023-07-12T15:21:50.913 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:21:50.914 INFO:teuthology.orchestra.run.smithi069.stdout:Total 6.5 MB/s | 3.4 MB 00:00 2023-07-12T15:21:50.914 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:21:50.926 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:21:50.927 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:21:50.993 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:21:50.997 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:21:51.431 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:21:51.607 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:21:51.973 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:21:52.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:21:52.983 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:19 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:56.184 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:21:56.368 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:56.368 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:21:56.368 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:21:56.369 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:56.369 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:21:56.489 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-nbd 2023-07-12T15:21:56.844 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-07-12T15:21:57.759 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:24 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:21:59.156 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-07-12T15:21:59.157 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:59.157 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2023-07-12T15:21:59.157 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:59.157 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2023-07-12T15:21:59.157 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd x86_64 2:18.1.2-55.gb1bd0540.el8 ceph 179 k 2023-07-12T15:21:59.158 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:59.158 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2023-07-12T15:21:59.158 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2023-07-12T15:21:59.158 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2023-07-12T15:21:59.158 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:21:59.158 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 179 k 2023-07-12T15:21:59.158 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 568 k 2023-07-12T15:21:59.159 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2023-07-12T15:21:59.341 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-nbd-18.1.2-55.gb1bd0540.el8.x86_64.rpm 978 kB/s | 179 kB 00:00 2023-07-12T15:21:59.343 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:21:59.343 INFO:teuthology.orchestra.run.smithi069.stdout:Total 972 kB/s | 179 kB 00:00 2023-07-12T15:21:59.343 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2023-07-12T15:21:59.353 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2023-07-12T15:21:59.354 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2023-07-12T15:21:59.407 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2023-07-12T15:21:59.411 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2023-07-12T15:21:59.562 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2023-07-12T15:21:59.720 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:21:59.952 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-nbd-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:22:00.996 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-2:18.1.2-55.gb1bd0540.el8.x86_64 1/1 2023-07-12T15:22:00.996 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:27 ago on Wed 12 Jul 2023 03:15:33 PM UTC. 2023-07-12T15:22:05.016 INFO:teuthology.orchestra.run.smithi069.stdout:Installed products updated. 2023-07-12T15:22:05.200 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:22:05.201 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2023-07-12T15:22:05.201 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-2:18.1.2-55.gb1bd0540.el8.x86_64 2023-07-12T15:22:05.201 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:22:05.202 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-07-12T15:22:05.333 DEBUG:teuthology.parallel:result is None 2023-07-12T15:22:05.334 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b1bd054042b7797ba5b70d145a1da31693dda494 2023-07-12T15:22:05.492 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-12T15:22:05.515 INFO:teuthology.orchestra.run.smithi069.stdout:18.1.2-55.gb1bd0540.el8 2023-07-12T15:22:05.516 INFO:teuthology.packaging:The installed version of ceph is 18.1.2-55.gb1bd0540.el8 2023-07-12T15:22:05.516 INFO:teuthology.task.install:The correct ceph version 18.1.2-55.gb1bd0540 is installed. 2023-07-12T15:22:05.517 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b1bd054042b7797ba5b70d145a1da31693dda494 2023-07-12T15:22:05.658 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-12T15:22:05.686 INFO:teuthology.orchestra.run.smithi145.stdout:18.1.2-55.gb1bd0540.el8 2023-07-12T15:22:05.686 INFO:teuthology.packaging:The installed version of ceph is 18.1.2-55.gb1bd0540.el8 2023-07-12T15:22:05.686 INFO:teuthology.task.install:The correct ceph version 18.1.2-55.gb1bd0540 is installed. 2023-07-12T15:22:05.687 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-12T15:22:05.687 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:22:05.688 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-12T15:22:05.715 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-07-12T15:22:05.716 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-12T15:22:05.759 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-12T15:22:05.760 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:22:05.760 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2023-07-12T15:22:05.786 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-12T15:22:05.852 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-07-12T15:22:05.852 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/daemon-helper 2023-07-12T15:22:05.882 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-12T15:22:05.955 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-12T15:22:05.956 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:22:05.956 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-12T15:22:05.982 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-12T15:22:06.048 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-07-12T15:22:06.048 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-12T15:22:06.077 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-12T15:22:06.145 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-12T15:22:06.146 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:22:06.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2023-07-12T15:22:06.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-12T15:22:06.238 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-07-12T15:22:06.239 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/stdin-killer 2023-07-12T15:22:06.266 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-12T15:22:06.333 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-12T15:22:06.424 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': 'b1bd054042b7797ba5b70d145a1da31693dda494', 'cephadm_mode': 'root'} 2023-07-12T15:22:06.424 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 2023-07-12T15:22:06.434 INFO:tasks.cephadm:Cluster fsid is dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:22:06.435 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-12T15:22:06.435 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.69', 'mon.c': '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]', 'mon.b': '172.21.15.145'} 2023-07-12T15:22:06.435 INFO:tasks.cephadm:First mon is mon.a on smithi069 2023-07-12T15:22:06.436 INFO:tasks.cephadm:First mgr is y 2023-07-12T15:22:06.436 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-12T15:22:06.436 DEBUG:teuthology.orchestra.run.smithi069:> sudo hostname $(hostname -s) 2023-07-12T15:22:06.464 DEBUG:teuthology.orchestra.run.smithi145:> sudo hostname $(hostname -s) 2023-07-12T15:22:06.493 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-12T15:22:06.493 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b1bd054042b7797ba5b70d145a1da31693dda494 2023-07-12T15:22:06.634 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'b1bd054042b7797ba5b70d145a1da31693dda494', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71337/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.1.2-55-gb1bd0540', 'node_name': '172.21.4.64+confusa02', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.1.2-55.gb1bd0540'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-guits-testing-4-2023-07-12-0746/b1bd054042b7797ba5b70d145a1da31693dda494/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-12 15:10:56.697946', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-testing-4-2023-07-12-0746', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-guits-testing-4-2023-07-12-0746/b1bd054042b7797ba5b70d145a1da31693dda494/centos/8/flavors/default/', 'archs': ['arm64', 'source', 'x86_64'], 'distro': 'centos'}] 2023-07-12T15:22:06.862 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-guits-testing-4-2023-07-12-0746, sha1 b1bd054042b7797ba5b70d145a1da31693dda494 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=b1bd054042b7797ba5b70d145a1da31693dda494 2023-07-12T15:22:06.863 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-guits-testing-4-2023-07-12-0746/b1bd054042b7797ba5b70d145a1da31693dda494/centos/8/x86_64/flavors/default/cephadm 2023-07-12T15:22:06.863 DEBUG:teuthology.orchestra.run.smithi069:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-guits-testing-4-2023-07-12-0746/b1bd054042b7797ba5b70d145a1da31693dda494/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-12T15:22:07.720 INFO:teuthology.orchestra.run.smithi069.stdout:-rw-rw-r--. 1 ubuntu ubuntu 648918 Jul 12 15:22 /home/ubuntu/cephtest/cephadm 2023-07-12T15:22:07.721 DEBUG:teuthology.orchestra.run.smithi145:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-guits-testing-4-2023-07-12-0746/b1bd054042b7797ba5b70d145a1da31693dda494/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-12T15:22:08.649 INFO:teuthology.orchestra.run.smithi145.stdout:-rw-rw-r--. 1 ubuntu ubuntu 648918 Jul 12 15:22 /home/ubuntu/cephtest/cephadm 2023-07-12T15:22:08.649 DEBUG:teuthology.orchestra.run.smithi069:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-12T15:22:08.671 DEBUG:teuthology.orchestra.run.smithi145:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-12T15:22:08.703 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 on all hosts... 2023-07-12T15:22:08.704 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 pull 2023-07-12T15:22:08.714 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 pull 2023-07-12T15:22:10.023 INFO:teuthology.orchestra.run.smithi145.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494... 2023-07-12T15:22:10.260 INFO:teuthology.orchestra.run.smithi069.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494... 2023-07-12T15:22:38.537 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2023-07-12T15:22:38.537 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_version": "ceph version 18.1.2-55-gb1bd0540 (b1bd054042b7797ba5b70d145a1da31693dda494) reef (rc)", 2023-07-12T15:22:38.538 INFO:teuthology.orchestra.run.smithi145.stdout: "image_id": "45bffa74f7bc71fbf78772581abbffd1413c1b6137ad1180e7a4e8a6f81c9386", 2023-07-12T15:22:38.538 INFO:teuthology.orchestra.run.smithi145.stdout: "repo_digests": [ 2023-07-12T15:22:38.538 INFO:teuthology.orchestra.run.smithi145.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c4be3de0cb572873b18bf4d7ff19efd4676dcd86f150f2b38e50b5e2946cf6ef" 2023-07-12T15:22:38.538 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2023-07-12T15:22:38.538 INFO:teuthology.orchestra.run.smithi145.stdout:} 2023-07-12T15:22:41.244 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-07-12T15:22:41.244 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 18.1.2-55-gb1bd0540 (b1bd054042b7797ba5b70d145a1da31693dda494) reef (rc)", 2023-07-12T15:22:41.244 INFO:teuthology.orchestra.run.smithi069.stdout: "image_id": "45bffa74f7bc71fbf78772581abbffd1413c1b6137ad1180e7a4e8a6f81c9386", 2023-07-12T15:22:41.244 INFO:teuthology.orchestra.run.smithi069.stdout: "repo_digests": [ 2023-07-12T15:22:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c4be3de0cb572873b18bf4d7ff19efd4676dcd86f150f2b38e50b5e2946cf6ef" 2023-07-12T15:22:41.245 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-07-12T15:22:41.245 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-07-12T15:22:41.262 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph 2023-07-12T15:22:41.291 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph 2023-07-12T15:22:41.340 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /etc/ceph 2023-07-12T15:22:41.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 777 /etc/ceph 2023-07-12T15:22:41.396 INFO:tasks.cephadm:Writing seed config... 2023-07-12T15:22:41.397 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-12T15:22:41.398 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-12T15:22:41.398 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-12T15:22:41.398 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2023-07-12T15:22:41.398 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2023-07-12T15:22:41.399 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2023-07-12T15:22:41.399 INFO:tasks.cephadm: override: [global] ms type = async 2023-07-12T15:22:41.399 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-12T15:22:41.399 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-12T15:22:41.399 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-12T15:22:41.400 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2023-07-12T15:22:41.400 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-12T15:22:41.400 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-12T15:22:41.400 INFO:tasks.cephadm: override: [osd] osd class default list = * 2023-07-12T15:22:41.401 INFO:tasks.cephadm: override: [osd] osd class load list = * 2023-07-12T15:22:41.401 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-12T15:22:41.401 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:22:41.402 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-12T15:22:41.423 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = dc75a490-20c7-11ee-9b31-001a4aab830c mon election default strategy = 3 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2023-07-12T15:22:41.424 DEBUG:teuthology.orchestra.run.smithi069:mon.a> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.a.service 2023-07-12T15:22:41.465 DEBUG:teuthology.orchestra.run.smithi069:mgr.y> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mgr.y.service 2023-07-12T15:22:41.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:-- Logs begin at Wed 2023-07-12 15:05:12 UTC. -- 2023-07-12T15:22:41.491 INFO:tasks.cephadm:Bootstrapping... 2023-07-12T15:22:41.491 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 -v bootstrap --fsid dc75a490-20c7-11ee-9b31-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.69 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-12T15:22:41.516 INFO:journalctl@ceph.mgr.y.smithi069.stdout:-- Logs begin at Wed 2023-07-12 15:05:12 UTC. -- 2023-07-12T15:22:41.647 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2023-07-12T15:22:41.647 INFO:teuthology.orchestra.run.smithi069.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494', '-v', 'bootstrap', '--fsid', 'dc75a490-20c7-11ee-9b31-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.69', '--skip-admin-label'] 2023-07-12T15:22:41.796 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stdout 4.1.1 2023-07-12T15:22:41.797 INFO:teuthology.orchestra.run.smithi069.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-12T15:22:41.798 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying podman|docker is present... 2023-07-12T15:22:41.939 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stdout 4.1.1 2023-07-12T15:22:41.939 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying lvm2 is present... 2023-07-12T15:22:41.940 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying time synchronization is in place... 2023-07-12T15:22:41.946 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-12T15:22:41.946 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-12T15:22:41.952 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-12T15:22:41.953 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2023-07-12T15:22:41.958 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout enabled 2023-07-12T15:22:41.964 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout active 2023-07-12T15:22:41.964 INFO:teuthology.orchestra.run.smithi069.stdout:Unit chronyd.service is enabled and running 2023-07-12T15:22:41.964 INFO:teuthology.orchestra.run.smithi069.stdout:Repeating the final host check... 2023-07-12T15:22:42.109 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stdout 4.1.1 2023-07-12T15:22:42.109 INFO:teuthology.orchestra.run.smithi069.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-12T15:22:42.110 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl is present 2023-07-12T15:22:42.110 INFO:teuthology.orchestra.run.smithi069.stdout:lvcreate is present 2023-07-12T15:22:42.115 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-12T15:22:42.115 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-12T15:22:42.121 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-12T15:22:42.122 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2023-07-12T15:22:42.127 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout enabled 2023-07-12T15:22:42.133 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout active 2023-07-12T15:22:42.133 INFO:teuthology.orchestra.run.smithi069.stdout:Unit chronyd.service is enabled and running 2023-07-12T15:22:42.133 INFO:teuthology.orchestra.run.smithi069.stdout:Host looks OK 2023-07-12T15:22:42.133 INFO:teuthology.orchestra.run.smithi069.stdout:Cluster fsid: dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:22:42.134 INFO:teuthology.orchestra.run.smithi069.stdout:Acquiring lock 140334462811944 on /run/cephadm/dc75a490-20c7-11ee-9b31-001a4aab830c.lock 2023-07-12T15:22:42.134 INFO:teuthology.orchestra.run.smithi069.stdout:Lock 140334462811944 acquired on /run/cephadm/dc75a490-20c7-11ee-9b31-001a4aab830c.lock 2023-07-12T15:22:42.134 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying IP 172.21.15.69 port 3300 ... 2023-07-12T15:22:42.135 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying IP 172.21.15.69 port 6789 ... 2023-07-12T15:22:42.135 INFO:teuthology.orchestra.run.smithi069.stdout:Base mon IP(s) is [172.21.15.69:3300, 172.21.15.69:6789], mon addrv is [v2:172.21.15.69:3300,v1:172.21.15.69:6789] 2023-07-12T15:22:42.137 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2023-07-12T15:22:42.137 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.69 metric 100 2023-07-12T15:22:42.140 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-12T15:22:42.140 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2023-07-12T15:22:42.140 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2023-07-12T15:22:42.143 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-12T15:22:42.143 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-12T15:22:42.144 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-12T15:22:42.144 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-07-12T15:22:42.144 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout inet6 fe80::fc63:4e94:c5cb:424a/64 scope link noprefixroute 2023-07-12T15:22:42.145 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-12T15:22:42.145 INFO:teuthology.orchestra.run.smithi069.stdout:Mon IP `172.21.15.69` is in CIDR network `172.21.0.0/20` 2023-07-12T15:22:42.145 INFO:teuthology.orchestra.run.smithi069.stdout:Mon IP `172.21.15.69` is in CIDR network `172.21.0.0/20` 2023-07-12T15:22:42.145 INFO:teuthology.orchestra.run.smithi069.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-12T15:22:42.146 INFO:teuthology.orchestra.run.smithi069.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-12T15:22:42.147 INFO:teuthology.orchestra.run.smithi069.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494... 2023-07-12T15:22:42.488 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stdout 45bffa74f7bc71fbf78772581abbffd1413c1b6137ad1180e7a4e8a6f81c9386 2023-07-12T15:22:42.489 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494... 2023-07-12T15:22:42.489 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stderr Getting image source signatures 2023-07-12T15:22:42.489 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stderr Copying blob sha256:fa1a69c5bf5b21be974af9bd78191a812fbc6592bc0eeda739702de591c5b12f 2023-07-12T15:22:42.489 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-12T15:22:42.489 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stderr Copying config sha256:45bffa74f7bc71fbf78772581abbffd1413c1b6137ad1180e7a4e8a6f81c9386 2023-07-12T15:22:42.489 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-12T15:22:42.490 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stderr Storing signatures 2023-07-12T15:22:43.340 INFO:teuthology.orchestra.run.smithi069.stdout:ceph: stdout ceph version 18.1.2-55-gb1bd0540 (b1bd054042b7797ba5b70d145a1da31693dda494) reef (rc) 2023-07-12T15:22:43.340 INFO:teuthology.orchestra.run.smithi069.stdout:Ceph version: ceph version 18.1.2-55-gb1bd0540 (b1bd054042b7797ba5b70d145a1da31693dda494) reef (rc) 2023-07-12T15:22:43.341 INFO:teuthology.orchestra.run.smithi069.stdout:Extracting ceph user uid/gid from container image... 2023-07-12T15:22:44.086 INFO:teuthology.orchestra.run.smithi069.stdout:stat: stdout 167 167 2023-07-12T15:22:44.086 INFO:teuthology.orchestra.run.smithi069.stdout:Creating initial keys... 2023-07-12T15:22:44.835 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-authtool: stdout AQBExa5kmgTQHBAAY3tADw7wpdOsNHffMBosXw== 2023-07-12T15:22:45.491 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-authtool: stdout AQBFxa5kVOkaDBAARrZmB+JZatfJRa8Z/xZS2Q== 2023-07-12T15:22:46.151 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-authtool: stdout AQBFxa5kqs6oMBAAeuKtsk643zAk3aMoNe5a4Q== 2023-07-12T15:22:46.151 INFO:teuthology.orchestra.run.smithi069.stdout:Creating initial monmap... 2023-07-12T15:22:46.906 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-12T15:22:46.907 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-12T15:22:46.907 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:22:46.907 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-12T15:22:46.907 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool for a [v2:172.21.15.69:3300,v1:172.21.15.69:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-12T15:22:46.907 INFO:teuthology.orchestra.run.smithi069.stdout:setting min_mon_release = pacific 2023-07-12T15:22:46.908 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: set fsid to dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:22:46.908 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-12T15:22:46.908 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:22:46.908 INFO:teuthology.orchestra.run.smithi069.stdout:Creating mon... 2023-07-12T15:22:47.756 INFO:teuthology.orchestra.run.smithi069.stdout:create mon.a on 2023-07-12T15:22:47.852 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-12T15:22:47.937 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-12T15:22:48.029 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-dc75a490-20c7-11ee-9b31-001a4aab830c.target → /etc/systemd/system/ceph-dc75a490-20c7-11ee-9b31-001a4aab830c.target. 2023-07-12T15:22:48.029 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-dc75a490-20c7-11ee-9b31-001a4aab830c.target → /etc/systemd/system/ceph-dc75a490-20c7-11ee-9b31-001a4aab830c.target. 2023-07-12T15:22:48.147 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.a 2023-07-12T15:22:48.147 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to reset failed state of unit ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.a.service: Unit ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.a.service not loaded. 2023-07-12T15:22:48.233 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-dc75a490-20c7-11ee-9b31-001a4aab830c.target.wants/ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@.service. 2023-07-12T15:22:48.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:48 smithi069 systemd[1]: Starting Ceph mon.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:22:48.738 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-12T15:22:48.738 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout disabled 2023-07-12T15:22:48.744 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-12T15:22:48.745 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2023-07-12T15:22:48.745 INFO:teuthology.orchestra.run.smithi069.stdout:firewalld.service is not enabled 2023-07-12T15:22:48.745 INFO:teuthology.orchestra.run.smithi069.stdout:Not possible to enable service . firewalld.service is not available 2023-07-12T15:22:48.745 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mon to start... 2023-07-12T15:22:48.746 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mon... 2023-07-12T15:22:48.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:48 smithi069 podman[93693]: 2023-07-12T15:22:48.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:48 smithi069 bash[93693]: 5c13a7fa67df2256acd2961e7c4f470ea91ebc0872e341e2583b4569d266a5a2 2023-07-12T15:22:48.781 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:48 smithi069 systemd[1]: Started Ceph mon.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:22:48.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:48 smithi069 ceph-mon[93734]: mkfs dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:22:48.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:48 smithi069 ceph-mon[93734]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-12T15:22:49.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout cluster: 2023-07-12T15:22:49.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout id: dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:22:49.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-12T15:22:49.699 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2023-07-12T15:22:49.699 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout services: 2023-07-12T15:22:49.699 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.515014s) 2023-07-12T15:22:49.699 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-12T15:22:49.699 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-12T15:22:49.700 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2023-07-12T15:22:49.700 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout data: 2023-07-12T15:22:49.700 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-12T15:22:49.700 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-12T15:22:49.700 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-12T15:22:49.700 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout pgs: 2023-07-12T15:22:49.701 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2023-07-12T15:22:49.701 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.205+0000 7f389d27c700 1 Processor -- start 2023-07-12T15:22:49.701 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.205+0000 7f389d27c700 1 -- start start 2023-07-12T15:22:49.701 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.205+0000 7f389d27c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 0x7f3898106f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:22:49.701 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.205+0000 7f389d27c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3898077d50 con 0x7f3898104b20 2023-07-12T15:22:49.702 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.205+0000 7f3896ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 0x7f3898106f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:22:49.702 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.205+0000 7f3896ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 0x7f3898106f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39516/0 (socket says 172.21.15.69:39516) 2023-07-12T15:22:49.702 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.205+0000 7f3896ffd700 1 -- 172.21.15.69:0/349014117 learned_addr learned my addr 172.21.15.69:0/349014117 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:22:49.702 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.206+0000 7f3896ffd700 1 -- 172.21.15.69:0/349014117 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3898077e90 con 0x7f3898104b20 2023-07-12T15:22:49.703 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.206+0000 7f3896ffd700 1 --2- 172.21.15.69:0/349014117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 0x7f3898106f00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f38880098a0 tx=0x7f3888009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c172372d5d56ce4 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:22:49.703 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.206+0000 7f3895ffb700 1 -- 172.21.15.69:0/349014117 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3888004160 con 0x7f3898104b20 2023-07-12T15:22:49.703 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.206+0000 7f3895ffb700 1 -- 172.21.15.69:0/349014117 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38880042c0 con 0x7f3898104b20 2023-07-12T15:22:49.703 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.206+0000 7f3895ffb700 1 -- 172.21.15.69:0/349014117 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38880045a0 con 0x7f3898104b20 2023-07-12T15:22:49.703 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.206+0000 7f389d27c700 1 -- 172.21.15.69:0/349014117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 msgr2=0x7f3898106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:22:49.703 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.206+0000 7f389d27c700 1 --2- 172.21.15.69:0/349014117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 0x7f3898106f00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f38880098a0 tx=0x7f3888009bb0 comp rx=0 tx=0).stop 2023-07-12T15:22:49.703 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.207+0000 7f389d27c700 1 -- 172.21.15.69:0/349014117 shutdown_connections 2023-07-12T15:22:49.704 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.207+0000 7f389d27c700 1 --2- 172.21.15.69:0/349014117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 0x7f3898106f00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:22:49.704 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.207+0000 7f389d27c700 1 -- 172.21.15.69:0/349014117 >> 172.21.15.69:0/349014117 conn(0x7f38980fe540 msgr2=0x7f3898100950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:22:49.704 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.207+0000 7f389d27c700 1 -- 172.21.15.69:0/349014117 shutdown_connections 2023-07-12T15:22:49.704 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.207+0000 7f389d27c700 1 -- 172.21.15.69:0/349014117 wait complete. 2023-07-12T15:22:49.704 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.207+0000 7f389d27c700 1 Processor -- start 2023-07-12T15:22:49.704 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.207+0000 7f389d27c700 1 -- start start 2023-07-12T15:22:49.704 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.207+0000 7f389d27c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 0x7f3898199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:22:49.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.208+0000 7f389d27c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3898199b90 con 0x7f3898104b20 2023-07-12T15:22:49.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.208+0000 7f3896ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 0x7f3898199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:22:49.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.208+0000 7f3896ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 0x7f3898199650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39518/0 (socket says 172.21.15.69:39518) 2023-07-12T15:22:49.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.208+0000 7f3896ffd700 1 -- 172.21.15.69:0/3753641463 learned_addr learned my addr 172.21.15.69:0/3753641463 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:22:49.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.208+0000 7f3896ffd700 1 -- 172.21.15.69:0/3753641463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3888009550 con 0x7f3898104b20 2023-07-12T15:22:49.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.208+0000 7f3896ffd700 1 --2- 172.21.15.69:0/3753641463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 0x7f3898199650 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3888004000 tx=0x7f3888004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:22:49.705 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.208+0000 7f387ffff700 1 -- 172.21.15.69:0/3753641463 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3888011630 con 0x7f3898104b20 2023-07-12T15:22:49.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.208+0000 7f387ffff700 1 -- 172.21.15.69:0/3753641463 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3888011790 con 0x7f3898104b20 2023-07-12T15:22:49.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.208+0000 7f389d27c700 1 -- 172.21.15.69:0/3753641463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3898199df0 con 0x7f3898104b20 2023-07-12T15:22:49.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.208+0000 7f387ffff700 1 -- 172.21.15.69:0/3753641463 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3888011a30 con 0x7f3898104b20 2023-07-12T15:22:49.706 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.208+0000 7f389d27c700 1 -- 172.21.15.69:0/3753641463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389819cbf0 con 0x7f3898104b20 2023-07-12T15:22:49.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.208+0000 7f387ffff700 1 -- 172.21.15.69:0/3753641463 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3888022070 con 0x7f3898104b20 2023-07-12T15:22:49.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.208+0000 7f387ffff700 1 -- 172.21.15.69:0/3753641463 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f388801c450 con 0x7f3898104b20 2023-07-12T15:22:49.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.209+0000 7f389d27c700 1 -- 172.21.15.69:0/3753641463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3898193650 con 0x7f3898104b20 2023-07-12T15:22:49.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.211+0000 7f387ffff700 1 -- 172.21.15.69:0/3753641463 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f388801b450 con 0x7f3898104b20 2023-07-12T15:22:49.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.281+0000 7f389d27c700 1 -- 172.21.15.69:0/3753641463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f389819a2f0 con 0x7f3898104b20 2023-07-12T15:22:49.707 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.281+0000 7f387ffff700 1 -- 172.21.15.69:0/3753641463 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f388801b5f0 con 0x7f3898104b20 2023-07-12T15:22:49.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.282+0000 7f389d27c700 1 -- 172.21.15.69:0/3753641463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 msgr2=0x7f3898199650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:22:49.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.282+0000 7f389d27c700 1 --2- 172.21.15.69:0/3753641463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 0x7f3898199650 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3888004000 tx=0x7f3888004750 comp rx=0 tx=0).stop 2023-07-12T15:22:49.708 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.282+0000 7f389d27c700 1 -- 172.21.15.69:0/3753641463 shutdown_connections 2023-07-12T15:22:49.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.282+0000 7f389d27c700 1 --2- 172.21.15.69:0/3753641463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3898104b20 0x7f3898199650 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:22:49.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.282+0000 7f389d27c700 1 -- 172.21.15.69:0/3753641463 >> 172.21.15.69:0/3753641463 conn(0x7f38980fe540 msgr2=0x7f38980ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:22:49.709 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.282+0000 7f389d27c700 1 -- 172.21.15.69:0/3753641463 shutdown_connections 2023-07-12T15:22:49.710 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:49.282+0000 7f389d27c700 1 -- 172.21.15.69:0/3753641463 wait complete. 2023-07-12T15:22:49.710 INFO:teuthology.orchestra.run.smithi069.stdout:mon is available 2023-07-12T15:22:49.710 INFO:teuthology.orchestra.run.smithi069.stdout:Assimilating anything we can from ceph.conf... 2023-07-12T15:22:49.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:49 smithi069 ceph-mon[93734]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-12T15:22:49.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:49 smithi069 ceph-mon[93734]: monmap e1: 1 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0]} removed_ranks: {} 2023-07-12T15:22:49.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:49 smithi069 ceph-mon[93734]: fsmap 2023-07-12T15:22:49.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:49 smithi069 ceph-mon[93734]: osdmap e1: 0 total, 0 up, 0 in 2023-07-12T15:22:49.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:49 smithi069 ceph-mon[93734]: mgrmap e1: no daemons active 2023-07-12T15:22:49.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:49 smithi069 ceph-mon[93734]: from='client.? 172.21.15.69:0/3753641463' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-12T15:22:50.633 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2023-07-12T15:22:50.637 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [global] 2023-07-12T15:22:50.638 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout fsid = dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:22:50.638 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.69:3300,v1:172.21.15.69:6789] 2023-07-12T15:22:50.638 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-12T15:22:50.638 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-12T15:22:50.639 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-12T15:22:50.639 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-12T15:22:50.639 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2023-07-12T15:22:50.639 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-12T15:22:50.639 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-12T15:22:50.639 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2023-07-12T15:22:50.639 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [osd] 2023-07-12T15:22:50.639 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-12T15:22:50.640 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-12T15:22:50.640 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.166+0000 7fdcd365a700 1 Processor -- start 2023-07-12T15:22:50.640 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.166+0000 7fdcd365a700 1 -- start start 2023-07-12T15:22:50.640 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.167+0000 7fdcd365a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 0x7fdccc102a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:22:50.640 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.167+0000 7fdcd365a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc103020 con 0x7fdccc102640 2023-07-12T15:22:50.640 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.167+0000 7fdcd13f6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 0x7fdccc102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:22:50.640 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.167+0000 7fdcd13f6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 0x7fdccc102a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39520/0 (socket says 172.21.15.69:39520) 2023-07-12T15:22:50.641 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.167+0000 7fdcd13f6700 1 -- 172.21.15.69:0/353048446 learned_addr learned my addr 172.21.15.69:0/353048446 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:22:50.641 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.167+0000 7fdcd13f6700 1 -- 172.21.15.69:0/353048446 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc103840 con 0x7fdccc102640 2023-07-12T15:22:50.641 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.167+0000 7fdcd13f6700 1 --2- 172.21.15.69:0/353048446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 0x7fdccc102a50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00098a0 tx=0x7fdcc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d040e20f6685e8c server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:22:50.641 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.168+0000 7fdcbffff700 1 -- 172.21.15.69:0/353048446 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdcc0004160 con 0x7fdccc102640 2023-07-12T15:22:50.641 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.168+0000 7fdcbffff700 1 -- 172.21.15.69:0/353048446 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc00042c0 con 0x7fdccc102640 2023-07-12T15:22:50.641 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.168+0000 7fdcbffff700 1 -- 172.21.15.69:0/353048446 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdcc00045a0 con 0x7fdccc102640 2023-07-12T15:22:50.641 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.168+0000 7fdcd365a700 1 -- 172.21.15.69:0/353048446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 msgr2=0x7fdccc102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:22:50.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.168+0000 7fdcd365a700 1 --2- 172.21.15.69:0/353048446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 0x7fdccc102a50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00098a0 tx=0x7fdcc0009bb0 comp rx=0 tx=0).stop 2023-07-12T15:22:50.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.168+0000 7fdcd365a700 1 -- 172.21.15.69:0/353048446 shutdown_connections 2023-07-12T15:22:50.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.168+0000 7fdcd365a700 1 --2- 172.21.15.69:0/353048446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 0x7fdccc102a50 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:22:50.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.168+0000 7fdcd365a700 1 -- 172.21.15.69:0/353048446 >> 172.21.15.69:0/353048446 conn(0x7fdccc0fdc30 msgr2=0x7fdccc100040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:22:50.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.168+0000 7fdcd365a700 1 -- 172.21.15.69:0/353048446 shutdown_connections 2023-07-12T15:22:50.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.168+0000 7fdcd365a700 1 -- 172.21.15.69:0/353048446 wait complete. 2023-07-12T15:22:50.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.169+0000 7fdcd365a700 1 Processor -- start 2023-07-12T15:22:50.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.169+0000 7fdcd365a700 1 -- start start 2023-07-12T15:22:50.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.169+0000 7fdcd365a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 0x7fdccc197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:22:50.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.169+0000 7fdcd365a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1979c0 con 0x7fdccc102640 2023-07-12T15:22:50.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.169+0000 7fdcd13f6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 0x7fdccc197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:22:50.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.169+0000 7fdcd13f6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 0x7fdccc197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39522/0 (socket says 172.21.15.69:39522) 2023-07-12T15:22:50.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.169+0000 7fdcd13f6700 1 -- 172.21.15.69:0/26326392 learned_addr learned my addr 172.21.15.69:0/26326392 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:22:50.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.169+0000 7fdcd13f6700 1 -- 172.21.15.69:0/26326392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc0009550 con 0x7fdccc102640 2023-07-12T15:22:50.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.169+0000 7fdcd13f6700 1 --2- 172.21.15.69:0/26326392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 0x7fdccc197480 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0004000 tx=0x7fdcc0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:22:50.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.170+0000 7fdcbe7fc700 1 -- 172.21.15.69:0/26326392 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdcc0011630 con 0x7fdccc102640 2023-07-12T15:22:50.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.170+0000 7fdcbe7fc700 1 -- 172.21.15.69:0/26326392 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc0011790 con 0x7fdccc102640 2023-07-12T15:22:50.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.170+0000 7fdcd365a700 1 -- 172.21.15.69:0/26326392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdccc197bc0 con 0x7fdccc102640 2023-07-12T15:22:50.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.170+0000 7fdcbe7fc700 1 -- 172.21.15.69:0/26326392 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdcc0011a30 con 0x7fdccc102640 2023-07-12T15:22:50.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.170+0000 7fdcd365a700 1 -- 172.21.15.69:0/26326392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdccc198060 con 0x7fdccc102640 2023-07-12T15:22:50.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.170+0000 7fdcbe7fc700 1 -- 172.21.15.69:0/26326392 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fdcc0022070 con 0x7fdccc102640 2023-07-12T15:22:50.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.170+0000 7fdcbe7fc700 1 -- 172.21.15.69:0/26326392 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdcc001c450 con 0x7fdccc102640 2023-07-12T15:22:50.645 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.171+0000 7fdcd365a700 1 -- 172.21.15.69:0/26326392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdccc062130 con 0x7fdccc102640 2023-07-12T15:22:50.645 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.173+0000 7fdcbe7fc700 1 -- 172.21.15.69:0/26326392 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fdcc001b450 con 0x7fdccc102640 2023-07-12T15:22:50.645 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.240+0000 7fdcd365a700 1 -- 172.21.15.69:0/26326392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fdccc19aa20 con 0x7fdccc102640 2023-07-12T15:22:50.645 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.243+0000 7fdcbe7fc700 1 -- 172.21.15.69:0/26326392 <== mon.0 v2:172.21.15.69:3300/0 7 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7fdcc001b5f0 con 0x7fdccc102640 2023-07-12T15:22:50.645 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.243+0000 7fdcbe7fc700 1 -- 172.21.15.69:0/26326392 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+338 (secure 0 0 0) 0x7fdcc00391c0 con 0x7fdccc102640 2023-07-12T15:22:50.645 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.244+0000 7fdcd365a700 1 -- 172.21.15.69:0/26326392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 msgr2=0x7fdccc197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:22:50.645 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.244+0000 7fdcd365a700 1 --2- 172.21.15.69:0/26326392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 0x7fdccc197480 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0004000 tx=0x7fdcc0004750 comp rx=0 tx=0).stop 2023-07-12T15:22:50.646 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.244+0000 7fdcd365a700 1 -- 172.21.15.69:0/26326392 shutdown_connections 2023-07-12T15:22:50.646 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.244+0000 7fdcd365a700 1 --2- 172.21.15.69:0/26326392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdccc102640 0x7fdccc197480 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:22:50.646 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.244+0000 7fdcd365a700 1 -- 172.21.15.69:0/26326392 >> 172.21.15.69:0/26326392 conn(0x7fdccc0fdc30 msgr2=0x7fdccc189770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:22:50.646 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.244+0000 7fdcd365a700 1 -- 172.21.15.69:0/26326392 shutdown_connections 2023-07-12T15:22:50.646 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:50.244+0000 7fdcd365a700 1 -- 172.21.15.69:0/26326392 wait complete. 2023-07-12T15:22:50.646 INFO:teuthology.orchestra.run.smithi069.stdout:Generating new minimal ceph.conf... 2023-07-12T15:22:50.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:50 smithi069 ceph-mon[93734]: from='client.? 172.21.15.69:0/26326392' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-12T15:22:50.913 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:50 smithi069 ceph-mon[93734]: from='client.? 172.21.15.69:0/26326392' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-12T15:22:51.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:51 smithi069 ceph-mon[93734]: from='client.? 172.21.15.69:0/907797728' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:22:52.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.139+0000 7f377db62700 1 Processor -- start 2023-07-12T15:22:52.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.139+0000 7f377db62700 1 -- start start 2023-07-12T15:22:52.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.139+0000 7f377db62700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 0x7f3778106e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:22:52.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.139+0000 7f377db62700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3778077e40 con 0x7f3778104a90 2023-07-12T15:22:52.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.139+0000 7f37777fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 0x7f3778106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:22:52.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.139+0000 7f37777fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 0x7f3778106e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39524/0 (socket says 172.21.15.69:39524) 2023-07-12T15:22:52.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.139+0000 7f37777fe700 1 -- 172.21.15.69:0/2801424025 learned_addr learned my addr 172.21.15.69:0/2801424025 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:22:52.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.140+0000 7f37777fe700 1 -- 172.21.15.69:0/2801424025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37781073b0 con 0x7f3778104a90 2023-07-12T15:22:52.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.140+0000 7f37777fe700 1 --2- 172.21.15.69:0/2801424025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 0x7f3778106e70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f376c0098a0 tx=0x7f376c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a2364680ab14fb server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:22:52.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.140+0000 7f37767fc700 1 -- 172.21.15.69:0/2801424025 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f376c004160 con 0x7f3778104a90 2023-07-12T15:22:52.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.140+0000 7f37767fc700 1 -- 172.21.15.69:0/2801424025 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f376c0042c0 con 0x7f3778104a90 2023-07-12T15:22:52.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.140+0000 7f37767fc700 1 -- 172.21.15.69:0/2801424025 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f376c011900 con 0x7f3778104a90 2023-07-12T15:22:52.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.140+0000 7f377db62700 1 -- 172.21.15.69:0/2801424025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 msgr2=0x7f3778106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:22:52.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.140+0000 7f377db62700 1 --2- 172.21.15.69:0/2801424025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 0x7f3778106e70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f376c0098a0 tx=0x7f376c009bb0 comp rx=0 tx=0).stop 2023-07-12T15:22:52.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.141+0000 7f377db62700 1 -- 172.21.15.69:0/2801424025 shutdown_connections 2023-07-12T15:22:52.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.141+0000 7f377db62700 1 --2- 172.21.15.69:0/2801424025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 0x7f3778106e70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:22:52.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.141+0000 7f377db62700 1 -- 172.21.15.69:0/2801424025 >> 172.21.15.69:0/2801424025 conn(0x7f37780fe410 msgr2=0x7f3778100880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:22:52.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.141+0000 7f377db62700 1 -- 172.21.15.69:0/2801424025 shutdown_connections 2023-07-12T15:22:52.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.141+0000 7f377db62700 1 -- 172.21.15.69:0/2801424025 wait complete. 2023-07-12T15:22:52.885 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.141+0000 7f377db62700 1 Processor -- start 2023-07-12T15:22:52.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.141+0000 7f377db62700 1 -- start start 2023-07-12T15:22:52.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.141+0000 7f377db62700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 0x7f3778197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:22:52.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.141+0000 7f377db62700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37781981c0 con 0x7f3778104a90 2023-07-12T15:22:52.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.142+0000 7f37777fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 0x7f3778197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:22:52.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.142+0000 7f37777fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 0x7f3778197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39526/0 (socket says 172.21.15.69:39526) 2023-07-12T15:22:52.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.142+0000 7f37777fe700 1 -- 172.21.15.69:0/907797728 learned_addr learned my addr 172.21.15.69:0/907797728 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:22:52.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.142+0000 7f37777fe700 1 -- 172.21.15.69:0/907797728 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c009550 con 0x7f3778104a90 2023-07-12T15:22:52.886 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.142+0000 7f37777fe700 1 --2- 172.21.15.69:0/907797728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 0x7f3778197c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f376c006af0 tx=0x7f376c011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:22:52.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.142+0000 7f3774ff9700 1 -- 172.21.15.69:0/907797728 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f376c011d80 con 0x7f3778104a90 2023-07-12T15:22:52.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.142+0000 7f377db62700 1 -- 172.21.15.69:0/907797728 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37781983c0 con 0x7f3778104a90 2023-07-12T15:22:52.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.142+0000 7f3774ff9700 1 -- 172.21.15.69:0/907797728 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f376c003b10 con 0x7f3778104a90 2023-07-12T15:22:52.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.142+0000 7f3774ff9700 1 -- 172.21.15.69:0/907797728 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f376c01add0 con 0x7f3778104a90 2023-07-12T15:22:52.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.142+0000 7f377db62700 1 -- 172.21.15.69:0/907797728 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3778198860 con 0x7f3778104a90 2023-07-12T15:22:52.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.142+0000 7f3774ff9700 1 -- 172.21.15.69:0/907797728 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f376c02d6f0 con 0x7f3778104a90 2023-07-12T15:22:52.887 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.142+0000 7f3774ff9700 1 -- 172.21.15.69:0/907797728 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f376c024c60 con 0x7f3778104a90 2023-07-12T15:22:52.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.143+0000 7f377db62700 1 -- 172.21.15.69:0/907797728 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3778062130 con 0x7f3778104a90 2023-07-12T15:22:52.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.145+0000 7f3774ff9700 1 -- 172.21.15.69:0/907797728 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f376c0243f0 con 0x7f3778104a90 2023-07-12T15:22:52.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.212+0000 7f377db62700 1 -- 172.21.15.69:0/907797728 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f377819b210 con 0x7f3778104a90 2023-07-12T15:22:52.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.212+0000 7f3774ff9700 1 -- 172.21.15.69:0/907797728 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f376c01a430 con 0x7f3778104a90 2023-07-12T15:22:52.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.213+0000 7f377db62700 1 -- 172.21.15.69:0/907797728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 msgr2=0x7f3778197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:22:52.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.213+0000 7f377db62700 1 --2- 172.21.15.69:0/907797728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 0x7f3778197c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f376c006af0 tx=0x7f376c011980 comp rx=0 tx=0).stop 2023-07-12T15:22:52.888 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.213+0000 7f377db62700 1 -- 172.21.15.69:0/907797728 shutdown_connections 2023-07-12T15:22:52.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.213+0000 7f377db62700 1 --2- 172.21.15.69:0/907797728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3778104a90 0x7f3778197c80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:22:52.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.213+0000 7f377db62700 1 -- 172.21.15.69:0/907797728 >> 172.21.15.69:0/907797728 conn(0x7f37780fe410 msgr2=0x7f37780ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:22:52.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.213+0000 7f377db62700 1 -- 172.21.15.69:0/907797728 shutdown_connections 2023-07-12T15:22:52.889 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:51.213+0000 7f377db62700 1 -- 172.21.15.69:0/907797728 wait complete. 2023-07-12T15:22:52.890 INFO:teuthology.orchestra.run.smithi069.stdout:Restarting the monitor... 2023-07-12T15:22:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:52 smithi069 systemd[1]: Stopping Ceph mon.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:22:53.964 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:53 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mon-a[93709]: 2023-07-12T15:22:53.693+0000 7f025b029700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:22:53.964 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:53 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mon-a[93709]: 2023-07-12T15:22:53.693+0000 7f025b029700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-12T15:22:54.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 bash[94065]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mon-a 2023-07-12T15:22:54.693 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.a.service: Succeeded. 2023-07-12T15:22:54.694 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 systemd[1]: Stopped Ceph mon.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:22:54.694 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 systemd[1]: Starting Ceph mon.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:22:54.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 podman[94200]: 2023-07-12T15:22:54.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: set uid:gid to 167:167 (ceph:ceph) 2023-07-12T15:22:54.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: ceph version 18.1.2-55-gb1bd0540 (b1bd054042b7797ba5b70d145a1da31693dda494) reef (rc), process ceph-mon, pid 7 2023-07-12T15:22:54.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: pidfile_write: ignore empty --pid-file 2023-07-12T15:22:54.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: load: jerasure load: lrc 2023-07-12T15:22:54.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: RocksDB version: 7.9.2 2023-07-12T15:22:54.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Git sha 0 2023-07-12T15:22:54.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Compile date 2023-07-12 12:18:34 2023-07-12T15:22:54.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: DB SUMMARY 2023-07-12T15:22:54.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: DB Session ID: SOZTJVDNWCBP7JHGPROW 2023-07-12T15:22:54.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: CURRENT file: CURRENT 2023-07-12T15:22:54.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: IDENTITY file: IDENTITY 2023-07-12T15:22:54.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-12T15:22:54.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-12T15:22:54.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 86650 ; 2023-07-12T15:22:54.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.error_if_exists: 0 2023-07-12T15:22:54.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.create_if_missing: 0 2023-07-12T15:22:54.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.paranoid_checks: 1 2023-07-12T15:22:54.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-12T15:22:54.962 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-12T15:22:54.962 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-12T15:22:54.962 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.env: 0x5555c3e6a720 2023-07-12T15:22:54.962 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.fs: PosixFileSystem 2023-07-12T15:22:54.962 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.info_log: 0x5555c71ab3e0 2023-07-12T15:22:54.962 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_file_opening_threads: 16 2023-07-12T15:22:54.962 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.statistics: (nil) 2023-07-12T15:22:54.963 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.use_fsync: 0 2023-07-12T15:22:54.963 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_log_file_size: 0 2023-07-12T15:22:54.963 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-12T15:22:54.963 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-12T15:22:54.963 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.keep_log_file_num: 1000 2023-07-12T15:22:54.963 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.recycle_log_file_num: 0 2023-07-12T15:22:54.963 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.allow_fallocate: 1 2023-07-12T15:22:54.963 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.allow_mmap_reads: 0 2023-07-12T15:22:54.964 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.allow_mmap_writes: 0 2023-07-12T15:22:54.964 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.use_direct_reads: 0 2023-07-12T15:22:54.964 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-12T15:22:54.964 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.create_missing_column_families: 0 2023-07-12T15:22:54.964 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.db_log_dir: 2023-07-12T15:22:54.964 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.wal_dir: 2023-07-12T15:22:54.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-12T15:22:54.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-12T15:22:54.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-12T15:22:54.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-12T15:22:54.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-12T15:22:54.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-12T15:22:54.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.advise_random_on_open: 1 2023-07-12T15:22:54.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.db_write_buffer_size: 0 2023-07-12T15:22:54.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.write_buffer_manager: 0x5555c643c5a0 2023-07-12T15:22:54.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-12T15:22:54.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-12T15:22:54.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-12T15:22:54.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.rate_limiter: (nil) 2023-07-12T15:22:54.966 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-12T15:22:54.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.wal_recovery_mode: 2 2023-07-12T15:22:54.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.enable_thread_tracking: 0 2023-07-12T15:22:54.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.enable_pipelined_write: 0 2023-07-12T15:22:54.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.unordered_write: 0 2023-07-12T15:22:54.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-12T15:22:54.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-12T15:22:54.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-12T15:22:54.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-12T15:22:54.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.row_cache: None 2023-07-12T15:22:54.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.wal_filter: None 2023-07-12T15:22:54.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-12T15:22:54.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.allow_ingest_behind: 0 2023-07-12T15:22:54.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.two_write_queues: 0 2023-07-12T15:22:54.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.manual_wal_flush: 0 2023-07-12T15:22:54.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.wal_compression: 0 2023-07-12T15:22:54.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.atomic_flush: 0 2023-07-12T15:22:54.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-12T15:22:54.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-12T15:22:54.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-12T15:22:54.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.log_readahead_size: 0 2023-07-12T15:22:54.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-12T15:22:54.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.best_efforts_recovery: 0 2023-07-12T15:22:54.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-12T15:22:54.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-12T15:22:54.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.allow_data_in_errors: 0 2023-07-12T15:22:54.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.db_host_id: __hostname__ 2023-07-12T15:22:54.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-12T15:22:54.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_background_jobs: 2 2023-07-12T15:22:54.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_background_compactions: -1 2023-07-12T15:22:54.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_subcompactions: 1 2023-07-12T15:22:54.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-12T15:22:54.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-12T15:22:54.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-12T15:22:54.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_total_wal_size: 0 2023-07-12T15:22:54.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-12T15:22:54.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-12T15:22:54.972 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-12T15:22:54.972 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-12T15:22:54.972 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_open_files: -1 2023-07-12T15:22:54.972 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.bytes_per_sync: 0 2023-07-12T15:22:54.972 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-12T15:22:54.972 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-12T15:22:54.972 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compaction_readahead_size: 0 2023-07-12T15:22:54.973 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_background_flushes: -1 2023-07-12T15:22:54.973 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Compression algorithms supported: 2023-07-12T15:22:54.973 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-12T15:22:54.973 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: kZSTD supported: 0 2023-07-12T15:22:54.973 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: kXpressCompression supported: 0 2023-07-12T15:22:54.973 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: kLZ4HCCompression supported: 1 2023-07-12T15:22:54.973 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: kZlibCompression supported: 1 2023-07-12T15:22:54.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: kSnappyCompression supported: 1 2023-07-12T15:22:54.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: kLZ4Compression supported: 1 2023-07-12T15:22:54.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: kBZip2Compression supported: 0 2023-07-12T15:22:54.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-12T15:22:54.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-12T15:22:54.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-12T15:22:54.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-12T15:22:54.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-12T15:22:54.975 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.merge_operator: 2023-07-12T15:22:54.975 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compaction_filter: None 2023-07-12T15:22:54.975 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compaction_filter_factory: None 2023-07-12T15:22:54.975 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.sst_partitioner_factory: None 2023-07-12T15:22:54.975 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-12T15:22:54.975 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-12T15:22:54.975 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5555c71ab500) 2023-07-12T15:22:54.976 INFO:journalctl@ceph.mon.a.smithi069.stdout: cache_index_and_filter_blocks: 1 2023-07-12T15:22:54.976 INFO:journalctl@ceph.mon.a.smithi069.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-12T15:22:54.976 INFO:journalctl@ceph.mon.a.smithi069.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-12T15:22:54.976 INFO:journalctl@ceph.mon.a.smithi069.stdout: pin_top_level_index_and_filter: 1 2023-07-12T15:22:54.976 INFO:journalctl@ceph.mon.a.smithi069.stdout: index_type: 0 2023-07-12T15:22:54.976 INFO:journalctl@ceph.mon.a.smithi069.stdout: data_block_index_type: 0 2023-07-12T15:22:54.976 INFO:journalctl@ceph.mon.a.smithi069.stdout: index_shortening: 1 2023-07-12T15:22:54.977 INFO:journalctl@ceph.mon.a.smithi069.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-12T15:22:54.977 INFO:journalctl@ceph.mon.a.smithi069.stdout: checksum: 4 2023-07-12T15:22:54.977 INFO:journalctl@ceph.mon.a.smithi069.stdout: no_block_cache: 0 2023-07-12T15:22:54.977 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache: 0x5555c64bd1f0 2023-07-12T15:22:54.978 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache_name: BinnedLRUCache 2023-07-12T15:22:54.978 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache_options: 2023-07-12T15:22:54.978 INFO:journalctl@ceph.mon.a.smithi069.stdout: capacity : 536870912 2023-07-12T15:22:54.978 INFO:journalctl@ceph.mon.a.smithi069.stdout: num_shard_bits : 4 2023-07-12T15:22:54.978 INFO:journalctl@ceph.mon.a.smithi069.stdout: strict_capacity_limit : 0 2023-07-12T15:22:54.978 INFO:journalctl@ceph.mon.a.smithi069.stdout: high_pri_pool_ratio: 0.000 2023-07-12T15:22:54.978 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache_compressed: (nil) 2023-07-12T15:22:54.979 INFO:journalctl@ceph.mon.a.smithi069.stdout: persistent_cache: (nil) 2023-07-12T15:22:54.979 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_size: 4096 2023-07-12T15:22:54.979 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_size_deviation: 10 2023-07-12T15:22:54.979 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_restart_interval: 16 2023-07-12T15:22:54.979 INFO:journalctl@ceph.mon.a.smithi069.stdout: index_block_restart_interval: 1 2023-07-12T15:22:54.979 INFO:journalctl@ceph.mon.a.smithi069.stdout: metadata_block_size: 4096 2023-07-12T15:22:54.979 INFO:journalctl@ceph.mon.a.smithi069.stdout: partition_filters: 0 2023-07-12T15:22:54.979 INFO:journalctl@ceph.mon.a.smithi069.stdout: use_delta_encoding: 1 2023-07-12T15:22:54.980 INFO:journalctl@ceph.mon.a.smithi069.stdout: filter_policy: bloomfilter 2023-07-12T15:22:54.980 INFO:journalctl@ceph.mon.a.smithi069.stdout: whole_key_filtering: 1 2023-07-12T15:22:54.980 INFO:journalctl@ceph.mon.a.smithi069.stdout: verify_compression: 0 2023-07-12T15:22:54.980 INFO:journalctl@ceph.mon.a.smithi069.stdout: read_amp_bytes_per_bit: 0 2023-07-12T15:22:54.980 INFO:journalctl@ceph.mon.a.smithi069.stdout: format_version: 5 2023-07-12T15:22:54.980 INFO:journalctl@ceph.mon.a.smithi069.stdout: enable_index_compression: 1 2023-07-12T15:22:54.980 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_align: 0 2023-07-12T15:22:54.980 INFO:journalctl@ceph.mon.a.smithi069.stdout: max_auto_readahead_size: 262144 2023-07-12T15:22:54.981 INFO:journalctl@ceph.mon.a.smithi069.stdout: prepopulate_block_cache: 0 2023-07-12T15:22:54.981 INFO:journalctl@ceph.mon.a.smithi069.stdout: initial_auto_readahead_size: 8192 2023-07-12T15:22:54.981 INFO:journalctl@ceph.mon.a.smithi069.stdout: num_file_reads_for_auto_readahead: 2 2023-07-12T15:22:54.981 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.write_buffer_size: 33554432 2023-07-12T15:22:54.981 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_write_buffer_number: 2 2023-07-12T15:22:54.981 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compression: NoCompression 2023-07-12T15:22:54.981 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.bottommost_compression: Disabled 2023-07-12T15:22:54.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.prefix_extractor: nullptr 2023-07-12T15:22:54.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-12T15:22:54.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.num_levels: 7 2023-07-12T15:22:54.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-12T15:22:54.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-12T15:22:54.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-12T15:22:54.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-12T15:22:54.982 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-12T15:22:54.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-12T15:22:54.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-12T15:22:54.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-12T15:22:54.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-12T15:22:54.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-12T15:22:54.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-12T15:22:54.983 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-12T15:22:54.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-12T15:22:54.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compression_opts.level: 32767 2023-07-12T15:22:54.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compression_opts.strategy: 0 2023-07-12T15:22:54.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-12T15:22:54.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-12T15:22:54.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-12T15:22:54.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-12T15:22:54.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compression_opts.enabled: false 2023-07-12T15:22:54.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-12T15:22:54.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-12T15:22:54.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-12T15:22:54.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-12T15:22:54.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.target_file_size_base: 67108864 2023-07-12T15:22:54.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-12T15:22:54.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-12T15:22:54.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-12T15:22:54.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-12T15:22:54.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-12T15:22:54.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-12T15:22:54.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-12T15:22:54.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-12T15:22:54.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-12T15:22:54.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-12T15:22:54.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-12T15:22:54.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-12T15:22:54.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-12T15:22:54.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-12T15:22:54.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.arena_block_size: 1048576 2023-07-12T15:22:54.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-12T15:22:54.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-12T15:22:54.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.disable_auto_compactions: 0 2023-07-12T15:22:54.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-12T15:22:54.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-12T15:22:54.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-12T15:22:54.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-12T15:22:54.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-12T15:22:54.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-12T15:22:54.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-12T15:22:54.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-12T15:22:54.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-12T15:22:54.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-12T15:22:54.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.table_properties_collectors: 2023-07-12T15:22:54.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.inplace_update_support: 0 2023-07-12T15:22:54.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-12T15:22:54.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-12T15:22:54.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-12T15:22:54.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-12T15:22:54.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.bloom_locality: 0 2023-07-12T15:22:54.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.max_successive_merges: 0 2023-07-12T15:22:54.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-12T15:22:54.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.paranoid_file_checks: 0 2023-07-12T15:22:54.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.force_consistency_checks: 1 2023-07-12T15:22:54.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.report_bg_io_stats: 0 2023-07-12T15:22:54.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.ttl: 2592000 2023-07-12T15:22:54.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-12T15:22:54.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-12T15:22:54.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-12T15:22:54.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.enable_blob_files: false 2023-07-12T15:22:54.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.min_blob_size: 0 2023-07-12T15:22:54.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.blob_file_size: 268435456 2023-07-12T15:22:54.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-12T15:22:54.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-12T15:22:54.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-12T15:22:54.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-12T15:22:54.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-12T15:22:54.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.blob_file_starting_level: 0 2023-07-12T15:22:54.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-12T15:22:54.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-12T15:22:54.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-12T15:22:54.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cb079a00-d87e-4016-8c9e-69bdf424d5d3 2023-07-12T15:22:54.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689175374884192, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-12T15:22:54.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-12T15:22:54.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689175374885062, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 83017, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 264, "table_properties": {"data_size": 81189, "index_size": 238, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10683, "raw_average_key_size": 47, "raw_value_size": 75175, "raw_average_value_size": 337, "num_data_blocks": 10, "num_entries": 223, "num_filter_entries": 223, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689175374, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cb079a00-d87e-4016-8c9e-69bdf424d5d3", "db_session_id": "SOZTJVDNWCBP7JHGPROW", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-12T15:22:54.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689175374885132, "job": 1, "event": "recovery_finished"} 2023-07-12T15:22:54.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-12T15:22:54.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-12T15:22:54.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5555c6558000 2023-07-12T15:22:54.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: DB pointer 0x5555c6544000 2023-07-12T15:22:54.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-12T15:22:54.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-12T15:22:54.995 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** DB Stats ** 2023-07-12T15:22:54.995 INFO:journalctl@ceph.mon.a.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-12T15:22:54.995 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-12T15:22:54.996 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-12T15:22:54.996 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-12T15:22:54.996 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-12T15:22:54.996 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-12T15:22:54.996 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-12T15:22:54.996 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2023-07-12T15:22:54.996 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** Compaction Stats [default] ** 2023-07-12T15:22:54.997 INFO:journalctl@ceph.mon.a.smithi069.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-12T15:22:54.997 INFO:journalctl@ceph.mon.a.smithi069.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-12T15:22:54.997 INFO:journalctl@ceph.mon.a.smithi069.stdout: L0 2/0 82.91 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-12T15:22:54.997 INFO:journalctl@ceph.mon.a.smithi069.stdout: Sum 2/0 82.91 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-12T15:22:54.997 INFO:journalctl@ceph.mon.a.smithi069.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-12T15:22:54.997 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2023-07-12T15:22:54.997 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** Compaction Stats [default] ** 2023-07-12T15:22:54.997 INFO:journalctl@ceph.mon.a.smithi069.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-12T15:22:54.998 INFO:journalctl@ceph.mon.a.smithi069.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-12T15:22:54.998 INFO:journalctl@ceph.mon.a.smithi069.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 147.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-12T15:22:54.998 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2023-07-12T15:22:54.998 INFO:journalctl@ceph.mon.a.smithi069.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-12T15:22:54.998 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2023-07-12T15:22:54.998 INFO:journalctl@ceph.mon.a.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-12T15:22:54.999 INFO:journalctl@ceph.mon.a.smithi069.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-12T15:22:54.999 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-12T15:22:54.999 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-12T15:22:54.999 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-12T15:22:54.999 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-12T15:22:54.999 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative compaction: 0.00 GB write, 27.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-12T15:22:54.999 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval compaction: 0.00 GB write, 27.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-12T15:22:54.999 INFO:journalctl@ceph.mon.a.smithi069.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-12T15:22:55.000 INFO:journalctl@ceph.mon.a.smithi069.stdout: Block cache BinnedLRUCache@0x5555c64bd1f0#7 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-12T15:22:55.000 INFO:journalctl@ceph.mon.a.smithi069.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2023-07-12T15:22:55.000 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2023-07-12T15:22:55.000 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-12T15:22:55.000 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: starting mon.a rank 0 at public addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] at bind addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:22:55.000 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: mon.a@-1(???) e1 preinit fsid dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:22:55.000 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: mon.a@-1(???).mds e1 new map 2023-07-12T15:22:55.001 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: mon.a@-1(???).mds e1 print_map 2023-07-12T15:22:55.001 INFO:journalctl@ceph.mon.a.smithi069.stdout: e1 2023-07-12T15:22:55.001 INFO:journalctl@ceph.mon.a.smithi069.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-12T15:22:55.001 INFO:journalctl@ceph.mon.a.smithi069.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-12T15:22:55.001 INFO:journalctl@ceph.mon.a.smithi069.stdout: legacy client fscid: -1 2023-07-12T15:22:55.001 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2023-07-12T15:22:55.001 INFO:journalctl@ceph.mon.a.smithi069.stdout: No filesystems configured 2023-07-12T15:22:55.001 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-12T15:22:55.002 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-12T15:22:55.002 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-12T15:22:55.002 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-12T15:22:55.002 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-12T15:22:55.002 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: expand_channel_meta expand map: {default=false} 2023-07-12T15:22:55.002 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: expand_channel_meta from 'false' to 'false' 2023-07-12T15:22:55.002 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: expand_channel_meta expanded map: {default=false} 2023-07-12T15:22:55.003 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: expand_channel_meta expand map: {default=info} 2023-07-12T15:22:55.003 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: expand_channel_meta from 'info' to 'info' 2023-07-12T15:22:55.003 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: expand_channel_meta expanded map: {default=info} 2023-07-12T15:22:55.003 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: expand_channel_meta expand map: {default=daemon} 2023-07-12T15:22:55.003 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-12T15:22:55.003 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: expand_channel_meta expanded map: {default=daemon} 2023-07-12T15:22:55.003 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: expand_channel_meta expand map: {default=debug} 2023-07-12T15:22:55.003 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: expand_channel_meta from 'debug' to 'debug' 2023-07-12T15:22:55.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: expand_channel_meta expanded map: {default=debug} 2023-07-12T15:22:55.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 bash[94200]: 9395c406f38e95c595e58a81501afd63548dfad77eda973a429659545908a15b 2023-07-12T15:22:55.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-12T15:22:55.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: monmap e1: 1 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0]} removed_ranks: {} 2023-07-12T15:22:55.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: fsmap 2023-07-12T15:22:55.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: osdmap e1: 0 total, 0 up, 0 in 2023-07-12T15:22:55.004 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 ceph-mon[94241]: mgrmap e1: no daemons active 2023-07-12T15:22:55.005 INFO:teuthology.orchestra.run.smithi069.stdout:Setting mon public_network to 172.21.0.0/20 2023-07-12T15:22:55.232 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:54 smithi069 systemd[1]: Started Ceph mon.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:22:55.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.443+0000 7fed4cffe700 1 Processor -- start 2023-07-12T15:22:55.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.443+0000 7fed4cffe700 1 -- start start 2023-07-12T15:22:55.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.444+0000 7fed4cffe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 0x7fed481048e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:22:55.838 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.444+0000 7fed4cffe700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed48104eb0 con 0x7fed481044d0 2023-07-12T15:22:55.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.444+0000 7fed4659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 0x7fed481048e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:22:55.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.444+0000 7fed4659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 0x7fed481048e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39528/0 (socket says 172.21.15.69:39528) 2023-07-12T15:22:55.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.444+0000 7fed4659c700 1 -- 172.21.15.69:0/3748873279 learned_addr learned my addr 172.21.15.69:0/3748873279 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:22:55.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.444+0000 7fed4659c700 1 -- 172.21.15.69:0/3748873279 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed48104ff0 con 0x7fed481044d0 2023-07-12T15:22:55.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.444+0000 7fed4659c700 1 --2- 172.21.15.69:0/3748873279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 0x7fed481048e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fed3c0098a0 tx=0x7fed3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1531611de4f8c410 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:22:55.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.445+0000 7fed4559a700 1 -- 172.21.15.69:0/3748873279 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed3c004160 con 0x7fed481044d0 2023-07-12T15:22:55.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.445+0000 7fed4559a700 1 -- 172.21.15.69:0/3748873279 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7fed3c0042c0 con 0x7fed481044d0 2023-07-12T15:22:55.839 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.445+0000 7fed4559a700 1 -- 172.21.15.69:0/3748873279 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed3c011900 con 0x7fed481044d0 2023-07-12T15:22:55.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.445+0000 7fed4cffe700 1 -- 172.21.15.69:0/3748873279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 msgr2=0x7fed481048e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:22:55.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.445+0000 7fed4cffe700 1 --2- 172.21.15.69:0/3748873279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 0x7fed481048e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fed3c0098a0 tx=0x7fed3c009bb0 comp rx=0 tx=0).stop 2023-07-12T15:22:55.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.445+0000 7fed4cffe700 1 -- 172.21.15.69:0/3748873279 shutdown_connections 2023-07-12T15:22:55.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.445+0000 7fed4cffe700 1 --2- 172.21.15.69:0/3748873279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 0x7fed481048e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:22:55.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.445+0000 7fed4cffe700 1 -- 172.21.15.69:0/3748873279 >> 172.21.15.69:0/3748873279 conn(0x7fed480fdc30 msgr2=0x7fed48100040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:22:55.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.445+0000 7fed4cffe700 1 -- 172.21.15.69:0/3748873279 shutdown_connections 2023-07-12T15:22:55.840 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.445+0000 7fed4cffe700 1 -- 172.21.15.69:0/3748873279 wait complete. 2023-07-12T15:22:55.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.446+0000 7fed4cffe700 1 Processor -- start 2023-07-12T15:22:55.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.446+0000 7fed4cffe700 1 -- start start 2023-07-12T15:22:55.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.446+0000 7fed4cffe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 0x7fed48199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:22:55.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.446+0000 7fed4cffe700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4819a410 con 0x7fed481044d0 2023-07-12T15:22:55.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.446+0000 7fed4659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 0x7fed48199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:22:55.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.446+0000 7fed4659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 0x7fed48199ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39530/0 (socket says 172.21.15.69:39530) 2023-07-12T15:22:55.841 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.446+0000 7fed4659c700 1 -- 172.21.15.69:0/1149949899 learned_addr learned my addr 172.21.15.69:0/1149949899 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:22:55.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.447+0000 7fed4659c700 1 -- 172.21.15.69:0/1149949899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c009550 con 0x7fed481044d0 2023-07-12T15:22:55.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.447+0000 7fed4659c700 1 --2- 172.21.15.69:0/1149949899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 0x7fed48199ed0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fed3c006af0 tx=0x7fed3c011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:22:55.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.447+0000 7fed377fe700 1 -- 172.21.15.69:0/1149949899 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed3c0118b0 con 0x7fed481044d0 2023-07-12T15:22:55.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.447+0000 7fed377fe700 1 -- 172.21.15.69:0/1149949899 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7fed3c003b10 con 0x7fed481044d0 2023-07-12T15:22:55.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.447+0000 7fed4cffe700 1 -- 172.21.15.69:0/1149949899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4819a610 con 0x7fed481044d0 2023-07-12T15:22:55.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.447+0000 7fed377fe700 1 -- 172.21.15.69:0/1149949899 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed3c024dd0 con 0x7fed481044d0 2023-07-12T15:22:55.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.447+0000 7fed4cffe700 1 -- 172.21.15.69:0/1149949899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4819aab0 con 0x7fed481044d0 2023-07-12T15:22:55.842 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.447+0000 7fed377fe700 1 -- 172.21.15.69:0/1149949899 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fed3c01a730 con 0x7fed481044d0 2023-07-12T15:22:55.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.447+0000 7fed377fe700 1 -- 172.21.15.69:0/1149949899 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fed3c02dd60 con 0x7fed481044d0 2023-07-12T15:22:55.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.448+0000 7fed4cffe700 1 -- 172.21.15.69:0/1149949899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed48193bc0 con 0x7fed481044d0 2023-07-12T15:22:55.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.450+0000 7fed377fe700 1 -- 172.21.15.69:0/1149949899 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fed3c0245b0 con 0x7fed481044d0 2023-07-12T15:22:55.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.516+0000 7fed4cffe700 1 -- 172.21.15.69:0/1149949899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fed4802cb90 con 0x7fed481044d0 2023-07-12T15:22:55.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.518+0000 7fed377fe700 1 -- 172.21.15.69:0/1149949899 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fed3c02d3f0 con 0x7fed481044d0 2023-07-12T15:22:55.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.519+0000 7fed4cffe700 1 -- 172.21.15.69:0/1149949899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 msgr2=0x7fed48199ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:22:55.843 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.519+0000 7fed4cffe700 1 --2- 172.21.15.69:0/1149949899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 0x7fed48199ed0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fed3c006af0 tx=0x7fed3c011ea0 comp rx=0 tx=0).stop 2023-07-12T15:22:55.844 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.519+0000 7fed4cffe700 1 -- 172.21.15.69:0/1149949899 shutdown_connections 2023-07-12T15:22:55.844 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.519+0000 7fed4cffe700 1 --2- 172.21.15.69:0/1149949899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed481044d0 0x7fed48199ed0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:22:55.844 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.519+0000 7fed4cffe700 1 -- 172.21.15.69:0/1149949899 >> 172.21.15.69:0/1149949899 conn(0x7fed480fdc30 msgr2=0x7fed48190d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:22:55.844 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.519+0000 7fed4cffe700 1 -- 172.21.15.69:0/1149949899 shutdown_connections 2023-07-12T15:22:55.844 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:55.519+0000 7fed4cffe700 1 -- 172.21.15.69:0/1149949899 wait complete. 2023-07-12T15:22:55.844 INFO:teuthology.orchestra.run.smithi069.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-12T15:22:55.844 INFO:teuthology.orchestra.run.smithi069.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-12T15:22:55.845 INFO:teuthology.orchestra.run.smithi069.stdout:Creating mgr... 2023-07-12T15:22:55.845 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying port 9283 ... 2023-07-12T15:22:55.845 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying port 8765 ... 2023-07-12T15:22:55.958 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mgr.y 2023-07-12T15:22:55.958 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to reset failed state of unit ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mgr.y.service: Unit ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mgr.y.service not loaded. 2023-07-12T15:22:56.047 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-dc75a490-20c7-11ee-9b31-001a4aab830c.target.wants/ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mgr.y.service → /etc/systemd/system/ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@.service. 2023-07-12T15:22:56.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:56 smithi069 systemd[1]: Starting Ceph mgr.y for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:22:56.612 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1149949899' entity='client.admin' 2023-07-12T15:22:56.613 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:56 smithi069 podman[94473]: 2023-07-12T15:22:56.651 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-12T15:22:56.652 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout disabled 2023-07-12T15:22:56.659 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-12T15:22:56.659 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2023-07-12T15:22:56.659 INFO:teuthology.orchestra.run.smithi069.stdout:firewalld.service is not enabled 2023-07-12T15:22:56.660 INFO:teuthology.orchestra.run.smithi069.stdout:Not possible to enable service . firewalld.service is not available 2023-07-12T15:22:56.667 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-12T15:22:56.667 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout disabled 2023-07-12T15:22:56.675 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-12T15:22:56.675 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2023-07-12T15:22:56.675 INFO:teuthology.orchestra.run.smithi069.stdout:firewalld.service is not enabled 2023-07-12T15:22:56.675 INFO:teuthology.orchestra.run.smithi069.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-12T15:22:56.675 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mgr to start... 2023-07-12T15:22:56.675 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mgr... 2023-07-12T15:22:56.887 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:56 smithi069 bash[94473]: 039e79eee440d4564a5d1aad3d0e2fdc68afc6cb432ed42321e2eb94f0e6081f 2023-07-12T15:22:56.887 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:56 smithi069 systemd[1]: Started Ceph mgr.y for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:22:56.887 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:56.726+0000 7f283a75a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-12T15:22:56.887 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:56.806+0000 7f283a75a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-12T15:22:57.176 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:56.899+0000 7f283a75a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-12T15:22:57.176 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:56.997+0000 7f283a75a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-12T15:22:57.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:57 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:57.175+0000 7f283a75a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-12T15:22:57.623 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2023-07-12T15:22:57.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2023-07-12T15:22:57.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsid": "dc75a490-20c7-11ee-9b31-001a4aab830c", 2023-07-12T15:22:57.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "health": { 2023-07-12T15:22:57.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-12T15:22:57.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-12T15:22:57.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-12T15:22:57.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:22:57.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-12T15:22:57.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-12T15:22:57.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 0 2023-07-12T15:22:57.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2023-07-12T15:22:57.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-12T15:22:57.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "a" 2023-07-12T15:22:57.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2023-07-12T15:22:57.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-07-12T15:22:57.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-12T15:22:57.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:22:57.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-12T15:22:57.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-12T15:22:57.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:22:57.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-12T15:22:57.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:22:57.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-12T15:22:57.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-12T15:22:57.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-12T15:22:57.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-12T15:22:57.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-12T15:22:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-12T15:22:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:22:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-12T15:22:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-12T15:22:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-12T15:22:57.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-12T15:22:57.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-12T15:22:57.645 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-12T15:22:57.645 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-12T15:22:57.645 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-12T15:22:57.645 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-12T15:22:57.646 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:22:57.646 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-12T15:22:57.646 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:22:57.646 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-12T15:22:57.646 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-12T15:22:57.646 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:22:57.647 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-12T15:22:57.647 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-12T15:22:57.647 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-12T15:22:57.647 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-12T15:22:57.647 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-12T15:22:57.647 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-12T15:22:57.647 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "restful" 2023-07-12T15:22:57.647 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2023-07-12T15:22:57.648 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-12T15:22:57.648 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:22:57.648 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-12T15:22:57.648 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:22:57.648 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modified": "2023-07-12T15:22:48.770004+0000", 2023-07-12T15:22:57.648 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-12T15:22:57.649 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:22:57.649 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-12T15:22:57.649 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2023-07-12T15:22:57.649 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.168+0000 7fa94e320700 1 Processor -- start 2023-07-12T15:22:57.649 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.168+0000 7fa94e320700 1 -- start start 2023-07-12T15:22:57.649 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.168+0000 7fa94e320700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 0x7fa948106e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:22:57.650 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.168+0000 7fa94e320700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa948077e80 con 0x7fa948104a50 2023-07-12T15:22:57.650 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.168+0000 7fa947fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 0x7fa948106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:22:57.650 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.168+0000 7fa947fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 0x7fa948106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39536/0 (socket says 172.21.15.69:39536) 2023-07-12T15:22:57.650 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.168+0000 7fa947fff700 1 -- 172.21.15.69:0/2026305335 learned_addr learned my addr 172.21.15.69:0/2026305335 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:22:57.650 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.169+0000 7fa947fff700 1 -- 172.21.15.69:0/2026305335 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa948107370 con 0x7fa948104a50 2023-07-12T15:22:57.650 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.169+0000 7fa947fff700 1 --2- 172.21.15.69:0/2026305335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 0x7fa948106e30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa9380098a0 tx=0x7fa938009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b0d34d713dbea32 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:22:57.650 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.169+0000 7fa946ffd700 1 -- 172.21.15.69:0/2026305335 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa938004160 con 0x7fa948104a50 2023-07-12T15:22:57.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.169+0000 7fa946ffd700 1 -- 172.21.15.69:0/2026305335 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7fa9380042c0 con 0x7fa948104a50 2023-07-12T15:22:57.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.169+0000 7fa946ffd700 1 -- 172.21.15.69:0/2026305335 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa938011900 con 0x7fa948104a50 2023-07-12T15:22:57.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.170+0000 7fa94e320700 1 -- 172.21.15.69:0/2026305335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 msgr2=0x7fa948106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:22:57.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.170+0000 7fa94e320700 1 --2- 172.21.15.69:0/2026305335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 0x7fa948106e30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa9380098a0 tx=0x7fa938009bb0 comp rx=0 tx=0).stop 2023-07-12T15:22:57.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.170+0000 7fa94e320700 1 -- 172.21.15.69:0/2026305335 shutdown_connections 2023-07-12T15:22:57.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.170+0000 7fa94e320700 1 --2- 172.21.15.69:0/2026305335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 0x7fa948106e30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:22:57.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.170+0000 7fa94e320700 1 -- 172.21.15.69:0/2026305335 >> 172.21.15.69:0/2026305335 conn(0x7fa9480fe410 msgr2=0x7fa948100880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:22:57.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.170+0000 7fa94e320700 1 -- 172.21.15.69:0/2026305335 shutdown_connections 2023-07-12T15:22:57.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.170+0000 7fa94e320700 1 -- 172.21.15.69:0/2026305335 wait complete. 2023-07-12T15:22:57.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.170+0000 7fa94e320700 1 Processor -- start 2023-07-12T15:22:57.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.170+0000 7fa94e320700 1 -- start start 2023-07-12T15:22:57.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.171+0000 7fa94e320700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 0x7fa9481043b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:22:57.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.171+0000 7fa94e320700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9481048f0 con 0x7fa948104a50 2023-07-12T15:22:57.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.171+0000 7fa947fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 0x7fa9481043b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:22:57.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.171+0000 7fa947fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 0x7fa9481043b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39538/0 (socket says 172.21.15.69:39538) 2023-07-12T15:22:57.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.171+0000 7fa947fff700 1 -- 172.21.15.69:0/2114321789 learned_addr learned my addr 172.21.15.69:0/2114321789 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:22:57.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.171+0000 7fa947fff700 1 -- 172.21.15.69:0/2114321789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa938009550 con 0x7fa948104a50 2023-07-12T15:22:57.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.171+0000 7fa947fff700 1 --2- 172.21.15.69:0/2114321789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 0x7fa9481043b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa938006af0 tx=0x7fa938011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:22:57.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.171+0000 7fa9457fa700 1 -- 172.21.15.69:0/2114321789 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa938011d80 con 0x7fa948104a50 2023-07-12T15:22:57.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.171+0000 7fa9457fa700 1 -- 172.21.15.69:0/2114321789 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7fa938003b10 con 0x7fa948104a50 2023-07-12T15:22:57.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.171+0000 7fa94e320700 1 -- 172.21.15.69:0/2114321789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa948102a00 con 0x7fa948104a50 2023-07-12T15:22:57.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.171+0000 7fa9457fa700 1 -- 172.21.15.69:0/2114321789 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa93801add0 con 0x7fa948104a50 2023-07-12T15:22:57.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.171+0000 7fa94e320700 1 -- 172.21.15.69:0/2114321789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa948102ea0 con 0x7fa948104a50 2023-07-12T15:22:57.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.172+0000 7fa9457fa700 1 -- 172.21.15.69:0/2114321789 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa93802d6f0 con 0x7fa948104a50 2023-07-12T15:22:57.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.172+0000 7fa9457fa700 1 -- 172.21.15.69:0/2114321789 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa938024cb0 con 0x7fa948104a50 2023-07-12T15:22:57.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.172+0000 7fa94e320700 1 -- 172.21.15.69:0/2114321789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa948062130 con 0x7fa948104a50 2023-07-12T15:22:57.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.174+0000 7fa9457fa700 1 -- 172.21.15.69:0/2114321789 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fa9380243f0 con 0x7fa948104a50 2023-07-12T15:22:57.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.246+0000 7fa94e320700 1 -- 172.21.15.69:0/2114321789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa948103270 con 0x7fa948104a50 2023-07-12T15:22:57.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.246+0000 7fa9457fa700 1 -- 172.21.15.69:0/2114321789 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fa93801a430 con 0x7fa948104a50 2023-07-12T15:22:57.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.247+0000 7fa94e320700 1 -- 172.21.15.69:0/2114321789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 msgr2=0x7fa9481043b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:22:57.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.247+0000 7fa94e320700 1 --2- 172.21.15.69:0/2114321789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 0x7fa9481043b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa938006af0 tx=0x7fa938011980 comp rx=0 tx=0).stop 2023-07-12T15:22:57.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.247+0000 7fa94e320700 1 -- 172.21.15.69:0/2114321789 shutdown_connections 2023-07-12T15:22:57.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.247+0000 7fa94e320700 1 --2- 172.21.15.69:0/2114321789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa948104a50 0x7fa9481043b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:22:57.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.247+0000 7fa94e320700 1 -- 172.21.15.69:0/2114321789 >> 172.21.15.69:0/2114321789 conn(0x7fa9480fe410 msgr2=0x7fa9480ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:22:57.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.247+0000 7fa94e320700 1 -- 172.21.15.69:0/2114321789 shutdown_connections 2023-07-12T15:22:57.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:22:57.247+0000 7fa94e320700 1 -- 172.21.15.69:0/2114321789 wait complete. 2023-07-12T15:22:57.656 INFO:teuthology.orchestra.run.smithi069.stdout:mgr not available, waiting (1/15)... 2023-07-12T15:22:57.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:22:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2114321789' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-12T15:22:57.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:57 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:57.497+0000 7f283a75a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-12T15:22:57.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:57 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:57.588+0000 7f283a75a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-12T15:22:57.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:57 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:57.721+0000 7f283a75a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-12T15:22:58.232 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:57 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:57.815+0000 7f283a75a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-12T15:22:58.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:58 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:58.231+0000 7f283a75a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-12T15:22:59.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:58 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:58.797+0000 7f283a75a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-12T15:22:59.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:58 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:58.964+0000 7f283a75a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-12T15:22:59.985 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:59 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:59.728+0000 7f283a75a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-12T15:22:59.985 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:59 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:59.818+0000 7f283a75a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-12T15:23:00.252 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:22:59 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:22:59.997+0000 7f283a75a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-12T15:23:00.523 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:00 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:00.286+0000 7f283a75a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-12T15:23:00.586 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2023-07-12T15:23:00.587 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2023-07-12T15:23:00.587 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsid": "dc75a490-20c7-11ee-9b31-001a4aab830c", 2023-07-12T15:23:00.587 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "health": { 2023-07-12T15:23:00.587 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-12T15:23:00.587 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-12T15:23:00.587 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-12T15:23:00.587 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:00.588 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-12T15:23:00.588 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-12T15:23:00.588 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 0 2023-07-12T15:23:00.588 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2023-07-12T15:23:00.588 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-12T15:23:00.588 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "a" 2023-07-12T15:23:00.588 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2023-07-12T15:23:00.589 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-07-12T15:23:00.589 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-12T15:23:00.589 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:23:00.589 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-12T15:23:00.589 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-12T15:23:00.589 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:00.589 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-12T15:23:00.589 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:23:00.590 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-12T15:23:00.590 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-12T15:23:00.590 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-12T15:23:00.595 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-12T15:23:00.596 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-12T15:23:00.596 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-12T15:23:00.596 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:00.596 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-12T15:23:00.596 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-12T15:23:00.597 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-12T15:23:00.597 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-12T15:23:00.597 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-12T15:23:00.597 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-12T15:23:00.597 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-12T15:23:00.597 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-12T15:23:00.597 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-12T15:23:00.597 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:00.598 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-12T15:23:00.598 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:23:00.598 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-12T15:23:00.598 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-12T15:23:00.598 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:00.598 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-12T15:23:00.598 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-12T15:23:00.599 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-12T15:23:00.599 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-12T15:23:00.599 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-12T15:23:00.599 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-12T15:23:00.599 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "restful" 2023-07-12T15:23:00.599 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2023-07-12T15:23:00.599 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-12T15:23:00.599 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:00.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-12T15:23:00.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:23:00.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modified": "2023-07-12T15:22:48.770004+0000", 2023-07-12T15:23:00.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-12T15:23:00.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:00.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-12T15:23:00.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2023-07-12T15:23:00.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.148+0000 7fdde11c6700 1 Processor -- start 2023-07-12T15:23:00.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.148+0000 7fdde11c6700 1 -- start start 2023-07-12T15:23:00.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.148+0000 7fdde11c6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 0x7fdddc106e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:00.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.148+0000 7fdde11c6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc077e80 con 0x7fdddc104a50 2023-07-12T15:23:00.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.149+0000 7fdddad9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 0x7fdddc106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:00.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.149+0000 7fdddad9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 0x7fdddc106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39540/0 (socket says 172.21.15.69:39540) 2023-07-12T15:23:00.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.149+0000 7fdddad9d700 1 -- 172.21.15.69:0/2952088524 learned_addr learned my addr 172.21.15.69:0/2952088524 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:00.601 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.149+0000 7fdddad9d700 1 -- 172.21.15.69:0/2952088524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc107370 con 0x7fdddc104a50 2023-07-12T15:23:00.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.149+0000 7fdddad9d700 1 --2- 172.21.15.69:0/2952088524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 0x7fdddc106e30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fddd00098a0 tx=0x7fddd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45860629ae1ae46 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:00.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.149+0000 7fddd9d9b700 1 -- 172.21.15.69:0/2952088524 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddd0004160 con 0x7fdddc104a50 2023-07-12T15:23:00.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.149+0000 7fddd9d9b700 1 -- 172.21.15.69:0/2952088524 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7fddd00042c0 con 0x7fdddc104a50 2023-07-12T15:23:00.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.149+0000 7fddd9d9b700 1 -- 172.21.15.69:0/2952088524 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddd0011900 con 0x7fdddc104a50 2023-07-12T15:23:00.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.150+0000 7fdde11c6700 1 -- 172.21.15.69:0/2952088524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 msgr2=0x7fdddc106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:00.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.150+0000 7fdde11c6700 1 --2- 172.21.15.69:0/2952088524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 0x7fdddc106e30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fddd00098a0 tx=0x7fddd0009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:00.602 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.150+0000 7fdde11c6700 1 -- 172.21.15.69:0/2952088524 shutdown_connections 2023-07-12T15:23:00.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.150+0000 7fdde11c6700 1 --2- 172.21.15.69:0/2952088524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 0x7fdddc106e30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:00.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.150+0000 7fdde11c6700 1 -- 172.21.15.69:0/2952088524 >> 172.21.15.69:0/2952088524 conn(0x7fdddc0fe410 msgr2=0x7fdddc100880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:00.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.150+0000 7fdde11c6700 1 -- 172.21.15.69:0/2952088524 shutdown_connections 2023-07-12T15:23:00.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.150+0000 7fdde11c6700 1 -- 172.21.15.69:0/2952088524 wait complete. 2023-07-12T15:23:00.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.151+0000 7fdde11c6700 1 Processor -- start 2023-07-12T15:23:00.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.151+0000 7fdde11c6700 1 -- start start 2023-07-12T15:23:00.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.151+0000 7fdde11c6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 0x7fdddc197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:00.603 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.151+0000 7fdde11c6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc198190 con 0x7fdddc104a50 2023-07-12T15:23:00.604 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.151+0000 7fdddad9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 0x7fdddc197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:00.604 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.151+0000 7fdddad9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 0x7fdddc197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39542/0 (socket says 172.21.15.69:39542) 2023-07-12T15:23:00.604 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.151+0000 7fdddad9d700 1 -- 172.21.15.69:0/2273034381 learned_addr learned my addr 172.21.15.69:0/2273034381 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:00.604 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.152+0000 7fdddad9d700 1 -- 172.21.15.69:0/2273034381 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd0009550 con 0x7fdddc104a50 2023-07-12T15:23:00.604 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.152+0000 7fdddad9d700 1 --2- 172.21.15.69:0/2273034381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 0x7fdddc197c50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fddd0006af0 tx=0x7fddd0011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:00.604 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.152+0000 7fddc3fff700 1 -- 172.21.15.69:0/2273034381 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddd0011d80 con 0x7fdddc104a50 2023-07-12T15:23:00.604 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.152+0000 7fddc3fff700 1 -- 172.21.15.69:0/2273034381 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7fddd0003b10 con 0x7fdddc104a50 2023-07-12T15:23:00.605 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.152+0000 7fdde11c6700 1 -- 172.21.15.69:0/2273034381 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc198390 con 0x7fdddc104a50 2023-07-12T15:23:00.605 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.152+0000 7fddc3fff700 1 -- 172.21.15.69:0/2273034381 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddd001add0 con 0x7fdddc104a50 2023-07-12T15:23:00.605 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.152+0000 7fdde11c6700 1 -- 172.21.15.69:0/2273034381 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc198830 con 0x7fdddc104a50 2023-07-12T15:23:00.605 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.152+0000 7fddc3fff700 1 -- 172.21.15.69:0/2273034381 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fddd002d6f0 con 0x7fdddc104a50 2023-07-12T15:23:00.605 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.152+0000 7fddc3fff700 1 -- 172.21.15.69:0/2273034381 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fddd0024cb0 con 0x7fdddc104a50 2023-07-12T15:23:00.605 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.153+0000 7fdde11c6700 1 -- 172.21.15.69:0/2273034381 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdddc062130 con 0x7fdddc104a50 2023-07-12T15:23:00.605 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.155+0000 7fddc3fff700 1 -- 172.21.15.69:0/2273034381 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fddd00243f0 con 0x7fdddc104a50 2023-07-12T15:23:00.605 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.227+0000 7fdde11c6700 1 -- 172.21.15.69:0/2273034381 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdddc19b2a0 con 0x7fdddc104a50 2023-07-12T15:23:00.606 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.227+0000 7fddc3fff700 1 -- 172.21.15.69:0/2273034381 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fddd001a430 con 0x7fdddc104a50 2023-07-12T15:23:00.606 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.228+0000 7fdde11c6700 1 -- 172.21.15.69:0/2273034381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 msgr2=0x7fdddc197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:00.606 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.228+0000 7fdde11c6700 1 --2- 172.21.15.69:0/2273034381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 0x7fdddc197c50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fddd0006af0 tx=0x7fddd0011980 comp rx=0 tx=0).stop 2023-07-12T15:23:00.606 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.228+0000 7fdde11c6700 1 -- 172.21.15.69:0/2273034381 shutdown_connections 2023-07-12T15:23:00.606 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.228+0000 7fdde11c6700 1 --2- 172.21.15.69:0/2273034381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc104a50 0x7fdddc197c50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:00.606 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.228+0000 7fdde11c6700 1 -- 172.21.15.69:0/2273034381 >> 172.21.15.69:0/2273034381 conn(0x7fdddc0fe410 msgr2=0x7fdddc0ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:00.606 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.228+0000 7fdde11c6700 1 -- 172.21.15.69:0/2273034381 shutdown_connections 2023-07-12T15:23:00.607 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:00.228+0000 7fdde11c6700 1 -- 172.21.15.69:0/2273034381 wait complete. 2023-07-12T15:23:00.607 INFO:teuthology.orchestra.run.smithi069.stdout:mgr not available, waiting (2/15)... 2023-07-12T15:23:00.892 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:00 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:00.522+0000 7f283a75a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-12T15:23:00.892 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:00 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:00.673+0000 7f283a75a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-12T15:23:01.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2273034381' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-12T15:23:01.236 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:00 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:00.932+0000 7f283a75a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-12T15:23:01.943 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:01 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:01.625+0000 7f283a75a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-12T15:23:01.946 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:01 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:01.754+0000 7f283a75a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-12T15:23:02.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:01 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:01.941+0000 7f283a75a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-12T15:23:02.236 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:02 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:02.054+0000 7f283a75a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-12T15:23:02.735 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:02 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:02.268+0000 7f283a75a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-12T15:23:03.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:03.165+0000 7f283a75a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-12T15:23:03.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-mon[94241]: Activating manager daemon y 2023-07-12T15:23:03.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-mon[94241]: mgrmap e2: y(active, starting, since 0.00318842s) 2023-07-12T15:23:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-mon[94241]: from='mgr.14100 172.21.15.69:0/1231282920' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-12T15:23:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-mon[94241]: from='mgr.14100 172.21.15.69:0/1231282920' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-12T15:23:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-mon[94241]: from='mgr.14100 172.21.15.69:0/1231282920' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-12T15:23:03.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-mon[94241]: from='mgr.14100 172.21.15.69:0/1231282920' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:23:03.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-mon[94241]: from='mgr.14100 172.21.15.69:0/1231282920' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-12T15:23:03.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2303856868' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-12T15:23:03.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-mon[94241]: Manager daemon y is now available 2023-07-12T15:23:03.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-mon[94241]: from='mgr.14100 172.21.15.69:0/1231282920' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-12T15:23:03.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-mon[94241]: from='mgr.14100 172.21.15.69:0/1231282920' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-12T15:23:03.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-mon[94241]: from='mgr.14100 172.21.15.69:0/1231282920' entity='mgr.y' 2023-07-12T15:23:03.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:03 smithi069 ceph-mon[94241]: from='mgr.14100 172.21.15.69:0/1231282920' entity='mgr.y' 2023-07-12T15:23:03.641 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2023-07-12T15:23:03.641 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2023-07-12T15:23:03.641 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsid": "dc75a490-20c7-11ee-9b31-001a4aab830c", 2023-07-12T15:23:03.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "health": { 2023-07-12T15:23:03.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-12T15:23:03.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-12T15:23:03.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-12T15:23:03.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:03.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-12T15:23:03.642 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-12T15:23:03.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 0 2023-07-12T15:23:03.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2023-07-12T15:23:03.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-12T15:23:03.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "a" 2023-07-12T15:23:03.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2023-07-12T15:23:03.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-07-12T15:23:03.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-12T15:23:03.643 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:23:03.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-12T15:23:03.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-12T15:23:03.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:03.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-12T15:23:03.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:23:03.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-12T15:23:03.644 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-12T15:23:03.650 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-12T15:23:03.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-12T15:23:03.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-12T15:23:03.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-12T15:23:03.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:03.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-12T15:23:03.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-12T15:23:03.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-12T15:23:03.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-12T15:23:03.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-12T15:23:03.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-12T15:23:03.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-12T15:23:03.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-12T15:23:03.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-12T15:23:03.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:03.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-12T15:23:03.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:23:03.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-12T15:23:03.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-12T15:23:03.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:03.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-12T15:23:03.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-12T15:23:03.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-12T15:23:03.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-12T15:23:03.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-12T15:23:03.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-12T15:23:03.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "restful" 2023-07-12T15:23:03.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2023-07-12T15:23:03.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-12T15:23:03.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:03.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-12T15:23:03.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:23:03.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modified": "2023-07-12T15:22:48.770004+0000", 2023-07-12T15:23:03.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-12T15:23:03.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:03.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-12T15:23:03.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2023-07-12T15:23:03.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.114+0000 7fb9f88ee700 1 Processor -- start 2023-07-12T15:23:03.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.114+0000 7fb9f88ee700 1 -- start start 2023-07-12T15:23:03.656 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.115+0000 7fb9f88ee700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 0x7fb9f41043d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:03.656 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.115+0000 7fb9f88ee700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f4104910 con 0x7fb9f4103fc0 2023-07-12T15:23:03.656 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.115+0000 7fb9f259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 0x7fb9f41043d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:03.656 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.115+0000 7fb9f259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 0x7fb9f41043d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39544/0 (socket says 172.21.15.69:39544) 2023-07-12T15:23:03.656 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.115+0000 7fb9f259c700 1 -- 172.21.15.69:0/1283012213 learned_addr learned my addr 172.21.15.69:0/1283012213 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:03.656 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.115+0000 7fb9f259c700 1 -- 172.21.15.69:0/1283012213 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f4104a50 con 0x7fb9f4103fc0 2023-07-12T15:23:03.656 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.115+0000 7fb9f259c700 1 --2- 172.21.15.69:0/1283012213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 0x7fb9f41043d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb9e40098a0 tx=0x7fb9e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef610e39dc75862 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:03.657 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.115+0000 7fb9f159a700 1 -- 172.21.15.69:0/1283012213 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9e4004160 con 0x7fb9f4103fc0 2023-07-12T15:23:03.657 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.115+0000 7fb9f159a700 1 -- 172.21.15.69:0/1283012213 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7fb9e40042c0 con 0x7fb9f4103fc0 2023-07-12T15:23:03.657 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.115+0000 7fb9f159a700 1 -- 172.21.15.69:0/1283012213 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9e4011900 con 0x7fb9f4103fc0 2023-07-12T15:23:03.657 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.116+0000 7fb9f88ee700 1 -- 172.21.15.69:0/1283012213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 msgr2=0x7fb9f41043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:03.657 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.116+0000 7fb9f88ee700 1 --2- 172.21.15.69:0/1283012213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 0x7fb9f41043d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb9e40098a0 tx=0x7fb9e4009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:03.657 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.116+0000 7fb9f88ee700 1 -- 172.21.15.69:0/1283012213 shutdown_connections 2023-07-12T15:23:03.657 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.116+0000 7fb9f88ee700 1 --2- 172.21.15.69:0/1283012213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 0x7fb9f41043d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:03.658 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.116+0000 7fb9f88ee700 1 -- 172.21.15.69:0/1283012213 >> 172.21.15.69:0/1283012213 conn(0x7fb9f40ff630 msgr2=0x7fb9f4101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:03.658 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.116+0000 7fb9f88ee700 1 -- 172.21.15.69:0/1283012213 shutdown_connections 2023-07-12T15:23:03.658 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.116+0000 7fb9f88ee700 1 -- 172.21.15.69:0/1283012213 wait complete. 2023-07-12T15:23:03.658 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.117+0000 7fb9f88ee700 1 Processor -- start 2023-07-12T15:23:03.658 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.117+0000 7fb9f88ee700 1 -- start start 2023-07-12T15:23:03.658 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.117+0000 7fb9f88ee700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 0x7fb9f4197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:03.658 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.117+0000 7fb9f88ee700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f4198150 con 0x7fb9f4103fc0 2023-07-12T15:23:03.658 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.117+0000 7fb9f259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 0x7fb9f4197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:03.659 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.117+0000 7fb9f259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 0x7fb9f4197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39546/0 (socket says 172.21.15.69:39546) 2023-07-12T15:23:03.659 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.117+0000 7fb9f259c700 1 -- 172.21.15.69:0/2303856868 learned_addr learned my addr 172.21.15.69:0/2303856868 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:03.659 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.117+0000 7fb9f259c700 1 -- 172.21.15.69:0/2303856868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e4009550 con 0x7fb9f4103fc0 2023-07-12T15:23:03.659 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.118+0000 7fb9f259c700 1 --2- 172.21.15.69:0/2303856868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 0x7fb9f4197c10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4006af0 tx=0x7fb9e4011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:03.659 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.118+0000 7fb9e37fe700 1 -- 172.21.15.69:0/2303856868 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9e4011970 con 0x7fb9f4103fc0 2023-07-12T15:23:03.659 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.118+0000 7fb9e37fe700 1 -- 172.21.15.69:0/2303856868 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7fb9e4003b10 con 0x7fb9f4103fc0 2023-07-12T15:23:03.659 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.118+0000 7fb9f88ee700 1 -- 172.21.15.69:0/2303856868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f4198350 con 0x7fb9f4103fc0 2023-07-12T15:23:03.660 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.118+0000 7fb9e37fe700 1 -- 172.21.15.69:0/2303856868 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9e4024da0 con 0x7fb9f4103fc0 2023-07-12T15:23:03.660 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.118+0000 7fb9f88ee700 1 -- 172.21.15.69:0/2303856868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f41987f0 con 0x7fb9f4103fc0 2023-07-12T15:23:03.660 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.118+0000 7fb9e37fe700 1 -- 172.21.15.69:0/2303856868 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb9e401a780 con 0x7fb9f4103fc0 2023-07-12T15:23:03.660 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.118+0000 7fb9e37fe700 1 -- 172.21.15.69:0/2303856868 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb9e402de30 con 0x7fb9f4103fc0 2023-07-12T15:23:03.660 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.119+0000 7fb9e17fa700 1 -- 172.21.15.69:0/2303856868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d80050e0 con 0x7fb9f4103fc0 2023-07-12T15:23:03.660 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.122+0000 7fb9e37fe700 1 -- 172.21.15.69:0/2303856868 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fb9e40245b0 con 0x7fb9f4103fc0 2023-07-12T15:23:03.660 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.180+0000 7fb9e37fe700 1 -- 172.21.15.69:0/2303856868 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 2) v1 ==== 44821+0+0 (secure 0 0 0) 0x7fb9e402d3f0 con 0x7fb9f4103fc0 2023-07-12T15:23:03.660 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.198+0000 7fb9e17fa700 1 -- 172.21.15.69:0/2303856868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb9d8005a80 con 0x7fb9f4103fc0 2023-07-12T15:23:03.661 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.198+0000 7fb9e37fe700 1 -- 172.21.15.69:0/2303856868 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fb9e4003880 con 0x7fb9f4103fc0 2023-07-12T15:23:03.661 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.199+0000 7fb9e17fa700 1 -- 172.21.15.69:0/2303856868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 msgr2=0x7fb9f4197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:03.661 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.199+0000 7fb9e17fa700 1 --2- 172.21.15.69:0/2303856868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 0x7fb9f4197c10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4006af0 tx=0x7fb9e4011e80 comp rx=0 tx=0).stop 2023-07-12T15:23:03.661 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.199+0000 7fb9e17fa700 1 -- 172.21.15.69:0/2303856868 shutdown_connections 2023-07-12T15:23:03.661 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.199+0000 7fb9e17fa700 1 --2- 172.21.15.69:0/2303856868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f4103fc0 0x7fb9f4197c10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:03.661 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.199+0000 7fb9e17fa700 1 -- 172.21.15.69:0/2303856868 >> 172.21.15.69:0/2303856868 conn(0x7fb9f40ff630 msgr2=0x7fb9f4100360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:03.661 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.199+0000 7fb9e17fa700 1 -- 172.21.15.69:0/2303856868 shutdown_connections 2023-07-12T15:23:03.662 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:03.199+0000 7fb9e17fa700 1 -- 172.21.15.69:0/2303856868 wait complete. 2023-07-12T15:23:03.662 INFO:teuthology.orchestra.run.smithi069.stdout:mgr not available, waiting (3/15)... 2023-07-12T15:23:04.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:04 smithi069 ceph-mon[94241]: from='mgr.14100 172.21.15.69:0/1231282920' entity='mgr.y' 2023-07-12T15:23:04.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:04 smithi069 ceph-mon[94241]: mgrmap e3: y(active, since 1.00597s) 2023-07-12T15:23:07.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:06 smithi069 ceph-mon[94241]: mgrmap e4: y(active, since 2s) 2023-07-12T15:23:08.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3832133744' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-12T15:23:11.001 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2023-07-12T15:23:11.002 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2023-07-12T15:23:11.002 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsid": "dc75a490-20c7-11ee-9b31-001a4aab830c", 2023-07-12T15:23:11.002 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "health": { 2023-07-12T15:23:11.002 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-12T15:23:11.002 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-12T15:23:11.002 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-12T15:23:11.002 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:11.003 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-12T15:23:11.004 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-12T15:23:11.004 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 0 2023-07-12T15:23:11.004 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2023-07-12T15:23:11.004 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-12T15:23:11.005 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "a" 2023-07-12T15:23:11.005 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2023-07-12T15:23:11.005 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2023-07-12T15:23:11.005 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-12T15:23:11.005 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:23:11.005 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-12T15:23:11.005 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-12T15:23:11.005 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:11.006 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-12T15:23:11.006 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:23:11.006 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-12T15:23:11.006 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-12T15:23:11.006 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-12T15:23:11.012 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-12T15:23:11.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-12T15:23:11.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-12T15:23:11.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:11.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-12T15:23:11.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-12T15:23:11.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-12T15:23:11.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-12T15:23:11.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-12T15:23:11.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-12T15:23:11.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-12T15:23:11.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-12T15:23:11.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-12T15:23:11.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:11.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-12T15:23:11.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:23:11.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-12T15:23:11.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-12T15:23:11.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:11.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-12T15:23:11.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-12T15:23:11.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-12T15:23:11.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-12T15:23:11.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-12T15:23:11.016 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-12T15:23:11.016 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "restful" 2023-07-12T15:23:11.016 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2023-07-12T15:23:11.016 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-12T15:23:11.016 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:11.016 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-12T15:23:11.016 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-12T15:23:11.017 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modified": "2023-07-12T15:22:48.770004+0000", 2023-07-12T15:23:11.017 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-12T15:23:11.017 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2023-07-12T15:23:11.017 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-12T15:23:11.017 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2023-07-12T15:23:11.017 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.886+0000 7f0fb1a94700 1 Processor -- start 2023-07-12T15:23:11.017 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.887+0000 7f0fb1a94700 1 -- start start 2023-07-12T15:23:11.017 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.887+0000 7f0fb1a94700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 0x7f0fac103260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:11.018 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.887+0000 7f0fb1a94700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac1037a0 con 0x7f0fac102e50 2023-07-12T15:23:11.018 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.887+0000 7f0fab7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 0x7f0fac103260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:11.018 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.887+0000 7f0fab7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 0x7f0fac103260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39572/0 (socket says 172.21.15.69:39572) 2023-07-12T15:23:11.018 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.887+0000 7f0fab7fe700 1 -- 172.21.15.69:0/1686756875 learned_addr learned my addr 172.21.15.69:0/1686756875 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:11.018 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.887+0000 7f0fab7fe700 1 -- 172.21.15.69:0/1686756875 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fac1038e0 con 0x7f0fac102e50 2023-07-12T15:23:11.018 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.887+0000 7f0fab7fe700 1 --2- 172.21.15.69:0/1686756875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 0x7f0fac103260 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0f940098a0 tx=0x7f0f94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2512857ab1a8ee3c server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:11.018 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.887+0000 7f0faa7fc700 1 -- 172.21.15.69:0/1686756875 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f94004160 con 0x7f0fac102e50 2023-07-12T15:23:11.019 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.887+0000 7f0faa7fc700 1 -- 172.21.15.69:0/1686756875 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f0f940042c0 con 0x7f0fac102e50 2023-07-12T15:23:11.019 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.887+0000 7f0faa7fc700 1 -- 172.21.15.69:0/1686756875 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f94011900 con 0x7f0fac102e50 2023-07-12T15:23:11.019 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.888+0000 7f0fb1a94700 1 -- 172.21.15.69:0/1686756875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 msgr2=0x7f0fac103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:11.019 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.888+0000 7f0fb1a94700 1 --2- 172.21.15.69:0/1686756875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 0x7f0fac103260 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0f940098a0 tx=0x7f0f94009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:11.019 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.888+0000 7f0fb1a94700 1 -- 172.21.15.69:0/1686756875 shutdown_connections 2023-07-12T15:23:11.019 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.888+0000 7f0fb1a94700 1 --2- 172.21.15.69:0/1686756875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 0x7f0fac103260 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:11.019 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.888+0000 7f0fb1a94700 1 -- 172.21.15.69:0/1686756875 >> 172.21.15.69:0/1686756875 conn(0x7f0fac0fe410 msgr2=0x7f0fac100880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:11.020 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.888+0000 7f0fb1a94700 1 -- 172.21.15.69:0/1686756875 shutdown_connections 2023-07-12T15:23:11.020 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.888+0000 7f0fb1a94700 1 -- 172.21.15.69:0/1686756875 wait complete. 2023-07-12T15:23:11.020 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.889+0000 7f0fb1a94700 1 Processor -- start 2023-07-12T15:23:11.020 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.889+0000 7f0fb1a94700 1 -- start start 2023-07-12T15:23:11.020 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.889+0000 7f0fb1a94700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 0x7f0fac077b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:11.020 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.889+0000 7f0fb1a94700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac19a9e0 con 0x7f0fac102e50 2023-07-12T15:23:11.020 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.889+0000 7f0fab7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 0x7f0fac077b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:11.020 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.889+0000 7f0fab7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 0x7f0fac077b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39574/0 (socket says 172.21.15.69:39574) 2023-07-12T15:23:11.021 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.889+0000 7f0fab7fe700 1 -- 172.21.15.69:0/3832133744 learned_addr learned my addr 172.21.15.69:0/3832133744 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:11.021 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.889+0000 7f0fab7fe700 1 -- 172.21.15.69:0/3832133744 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f94009550 con 0x7f0fac102e50 2023-07-12T15:23:11.021 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.889+0000 7f0fab7fe700 1 --2- 172.21.15.69:0/3832133744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 0x7f0fac077b60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0f94006af0 tx=0x7f0f94011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:11.021 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.890+0000 7f0fa8ff9700 1 -- 172.21.15.69:0/3832133744 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f94011e40 con 0x7f0fac102e50 2023-07-12T15:23:11.021 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.890+0000 7f0fa8ff9700 1 -- 172.21.15.69:0/3832133744 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f0f94003b10 con 0x7f0fac102e50 2023-07-12T15:23:11.021 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.890+0000 7f0fb1a94700 1 -- 172.21.15.69:0/3832133744 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fac19ab20 con 0x7f0fac102e50 2023-07-12T15:23:11.021 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.890+0000 7f0fa8ff9700 1 -- 172.21.15.69:0/3832133744 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f94024e20 con 0x7f0fac102e50 2023-07-12T15:23:11.022 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.890+0000 7f0fb1a94700 1 -- 172.21.15.69:0/3832133744 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fac19afc0 con 0x7f0fac102e50 2023-07-12T15:23:11.022 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.890+0000 7f0fa8ff9700 1 -- 172.21.15.69:0/3832133744 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 4) v1 ==== 45019+0+0 (secure 0 0 0) 0x7f0f9401a430 con 0x7f0fac102e50 2023-07-12T15:23:11.022 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.891+0000 7f0fb1a94700 1 -- 172.21.15.69:0/3832133744 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fac062130 con 0x7f0fac102e50 2023-07-12T15:23:11.022 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.891+0000 7f0fa8ff9700 1 --2- 172.21.15.69:0/3832133744 >> v2:172.21.15.69:6800/654231506 conn(0x7f0f98038330 0x7f0f9803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:11.022 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.891+0000 7f0fa8ff9700 1 -- 172.21.15.69:0/3832133744 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0f9404bfd0 con 0x7f0fac102e50 2023-07-12T15:23:11.022 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.891+0000 7f0faaffd700 1 --2- 172.21.15.69:0/3832133744 >> v2:172.21.15.69:6800/654231506 conn(0x7f0f98038330 0x7f0f9803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:11.022 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.891+0000 7f0faaffd700 1 --2- 172.21.15.69:0/3832133744 >> v2:172.21.15.69:6800/654231506 conn(0x7f0f98038330 0x7f0f9803a7e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c006fd0 tx=0x7f0f9c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:11.022 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:07.896+0000 7f0fa8ff9700 1 -- 172.21.15.69:0/3832133744 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0f9401a6e0 con 0x7f0fac102e50 2023-07-12T15:23:11.023 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:08.155+0000 7f0fb1a94700 1 -- 172.21.15.69:0/3832133744 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0fac19b7e0 con 0x7f0fac102e50 2023-07-12T15:23:11.023 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:08.156+0000 7f0fa8ff9700 1 -- 172.21.15.69:0/3832133744 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f0f9402c3a0 con 0x7f0fac102e50 2023-07-12T15:23:11.023 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:08.158+0000 7f0fb1a94700 1 -- 172.21.15.69:0/3832133744 >> v2:172.21.15.69:6800/654231506 conn(0x7f0f98038330 msgr2=0x7f0f9803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:11.023 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:08.158+0000 7f0fb1a94700 1 --2- 172.21.15.69:0/3832133744 >> v2:172.21.15.69:6800/654231506 conn(0x7f0f98038330 0x7f0f9803a7e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c006fd0 tx=0x7f0f9c006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:11.023 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:08.158+0000 7f0fb1a94700 1 -- 172.21.15.69:0/3832133744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 msgr2=0x7f0fac077b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:11.023 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:08.158+0000 7f0fb1a94700 1 --2- 172.21.15.69:0/3832133744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 0x7f0fac077b60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0f94006af0 tx=0x7f0f94011980 comp rx=0 tx=0).stop 2023-07-12T15:23:11.023 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:08.158+0000 7f0fb1a94700 1 -- 172.21.15.69:0/3832133744 shutdown_connections 2023-07-12T15:23:11.024 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:08.158+0000 7f0fb1a94700 1 --2- 172.21.15.69:0/3832133744 >> v2:172.21.15.69:6800/654231506 conn(0x7f0f98038330 0x7f0f9803a7e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:11.024 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:08.158+0000 7f0fb1a94700 1 --2- 172.21.15.69:0/3832133744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fac102e50 0x7f0fac077b60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:11.024 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:08.158+0000 7f0fb1a94700 1 -- 172.21.15.69:0/3832133744 >> 172.21.15.69:0/3832133744 conn(0x7f0fac0fe410 msgr2=0x7f0fac0ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:11.024 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:08.158+0000 7f0fb1a94700 1 -- 172.21.15.69:0/3832133744 shutdown_connections 2023-07-12T15:23:11.024 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:08.158+0000 7f0fb1a94700 1 -- 172.21.15.69:0/3832133744 wait complete. 2023-07-12T15:23:11.024 INFO:teuthology.orchestra.run.smithi069.stdout:mgr is available 2023-07-12T15:23:13.866 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2023-07-12T15:23:13.866 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [global] 2023-07-12T15:23:13.866 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout fsid = dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:23:13.866 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-12T15:23:13.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-12T15:23:13.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-12T15:23:13.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-12T15:23:13.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2023-07-12T15:23:13.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-12T15:23:13.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-12T15:23:13.867 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2023-07-12T15:23:13.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [osd] 2023-07-12T15:23:13.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-12T15:23:13.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-12T15:23:13.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.264+0000 7f0a3ea0e700 1 Processor -- start 2023-07-12T15:23:13.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.264+0000 7f0a3ea0e700 1 -- start start 2023-07-12T15:23:13.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.264+0000 7f0a3ea0e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 0x7f0a381048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:13.868 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.264+0000 7f0a3ea0e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a38104e80 con 0x7f0a381044a0 2023-07-12T15:23:13.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.265+0000 7f0a37fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 0x7f0a381048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:13.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.265+0000 7f0a37fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 0x7f0a381048b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39578/0 (socket says 172.21.15.69:39578) 2023-07-12T15:23:13.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.265+0000 7f0a37fff700 1 -- 172.21.15.69:0/3449648469 learned_addr learned my addr 172.21.15.69:0/3449648469 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:13.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.265+0000 7f0a37fff700 1 -- 172.21.15.69:0/3449648469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a381030b0 con 0x7f0a381044a0 2023-07-12T15:23:13.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.265+0000 7f0a37fff700 1 --2- 172.21.15.69:0/3449648469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 0x7f0a381048b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0098a0 tx=0x7f0a2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa2b455f555e28 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:13.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.265+0000 7f0a36ffd700 1 -- 172.21.15.69:0/3449648469 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a2c004160 con 0x7f0a381044a0 2023-07-12T15:23:13.869 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.265+0000 7f0a36ffd700 1 -- 172.21.15.69:0/3449648469 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f0a2c0042c0 con 0x7f0a381044a0 2023-07-12T15:23:13.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.265+0000 7f0a36ffd700 1 -- 172.21.15.69:0/3449648469 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a2c011900 con 0x7f0a381044a0 2023-07-12T15:23:13.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.266+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/3449648469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 msgr2=0x7f0a381048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:13.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.266+0000 7f0a3ea0e700 1 --2- 172.21.15.69:0/3449648469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 0x7f0a381048b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0098a0 tx=0x7f0a2c009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:13.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.266+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/3449648469 shutdown_connections 2023-07-12T15:23:13.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.266+0000 7f0a3ea0e700 1 --2- 172.21.15.69:0/3449648469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 0x7f0a381048b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:13.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.266+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/3449648469 >> 172.21.15.69:0/3449648469 conn(0x7f0a380fdc30 msgr2=0x7f0a38100040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:13.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.266+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/3449648469 shutdown_connections 2023-07-12T15:23:13.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.266+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/3449648469 wait complete. 2023-07-12T15:23:13.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.266+0000 7f0a3ea0e700 1 Processor -- start 2023-07-12T15:23:13.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.267+0000 7f0a3ea0e700 1 -- start start 2023-07-12T15:23:13.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.267+0000 7f0a3ea0e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 0x7f0a38196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:13.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.267+0000 7f0a3ea0e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a381970e0 con 0x7f0a381044a0 2023-07-12T15:23:13.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.267+0000 7f0a37fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 0x7f0a38196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:13.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.267+0000 7f0a37fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 0x7f0a38196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39580/0 (socket says 172.21.15.69:39580) 2023-07-12T15:23:13.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.267+0000 7f0a37fff700 1 -- 172.21.15.69:0/1168807195 learned_addr learned my addr 172.21.15.69:0/1168807195 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:13.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.267+0000 7f0a37fff700 1 -- 172.21.15.69:0/1168807195 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2c009550 con 0x7f0a381044a0 2023-07-12T15:23:13.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.267+0000 7f0a37fff700 1 --2- 172.21.15.69:0/1168807195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 0x7f0a38196ba0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c006af0 tx=0x7f0a2c011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:13.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.268+0000 7f0a357fa700 1 -- 172.21.15.69:0/1168807195 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a2c0118b0 con 0x7f0a381044a0 2023-07-12T15:23:13.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.268+0000 7f0a357fa700 1 -- 172.21.15.69:0/1168807195 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f0a2c003b10 con 0x7f0a381044a0 2023-07-12T15:23:13.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.268+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/1168807195 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a381972e0 con 0x7f0a381044a0 2023-07-12T15:23:13.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.268+0000 7f0a357fa700 1 -- 172.21.15.69:0/1168807195 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a2c024dd0 con 0x7f0a381044a0 2023-07-12T15:23:13.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.268+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/1168807195 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a38197780 con 0x7f0a381044a0 2023-07-12T15:23:13.874 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.268+0000 7f0a357fa700 1 -- 172.21.15.69:0/1168807195 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 4) v1 ==== 45019+0+0 (secure 0 0 0) 0x7f0a2c01a430 con 0x7f0a381044a0 2023-07-12T15:23:13.874 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.269+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/1168807195 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a38062130 con 0x7f0a381044a0 2023-07-12T15:23:13.874 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.269+0000 7f0a357fa700 1 --2- 172.21.15.69:0/1168807195 >> v2:172.21.15.69:6800/654231506 conn(0x7f0a20038340 0x7f0a2003a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:13.874 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.269+0000 7f0a357fa700 1 -- 172.21.15.69:0/1168807195 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0a2c04bd30 con 0x7f0a381044a0 2023-07-12T15:23:13.874 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.269+0000 7f0a377fe700 1 --2- 172.21.15.69:0/1168807195 >> v2:172.21.15.69:6800/654231506 conn(0x7f0a20038340 0x7f0a2003a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:13.874 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.269+0000 7f0a377fe700 1 --2- 172.21.15.69:0/1168807195 >> v2:172.21.15.69:6800/654231506 conn(0x7f0a20038340 0x7f0a2003a7f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0a28006fd0 tx=0x7f0a28006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:13.874 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.273+0000 7f0a357fa700 1 -- 172.21.15.69:0/1168807195 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0a2c02d3f0 con 0x7f0a381044a0 2023-07-12T15:23:13.875 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.463+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/1168807195 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f0a381072f0 con 0x7f0a381044a0 2023-07-12T15:23:13.875 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.464+0000 7f0a357fa700 1 -- 172.21.15.69:0/1168807195 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+282 (secure 0 0 0) 0x7f0a2c02c3a0 con 0x7f0a381044a0 2023-07-12T15:23:13.875 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.466+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/1168807195 >> v2:172.21.15.69:6800/654231506 conn(0x7f0a20038340 msgr2=0x7f0a2003a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:13.875 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.466+0000 7f0a3ea0e700 1 --2- 172.21.15.69:0/1168807195 >> v2:172.21.15.69:6800/654231506 conn(0x7f0a20038340 0x7f0a2003a7f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0a28006fd0 tx=0x7f0a28006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:13.875 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.466+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/1168807195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 msgr2=0x7f0a38196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:13.875 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.466+0000 7f0a3ea0e700 1 --2- 172.21.15.69:0/1168807195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 0x7f0a38196ba0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c006af0 tx=0x7f0a2c011ea0 comp rx=0 tx=0).stop 2023-07-12T15:23:13.875 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.466+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/1168807195 shutdown_connections 2023-07-12T15:23:13.876 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.466+0000 7f0a3ea0e700 1 --2- 172.21.15.69:0/1168807195 >> v2:172.21.15.69:6800/654231506 conn(0x7f0a20038340 0x7f0a2003a7f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:13.876 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.466+0000 7f0a3ea0e700 1 --2- 172.21.15.69:0/1168807195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a381044a0 0x7f0a38196ba0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:13.876 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.466+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/1168807195 >> 172.21.15.69:0/1168807195 conn(0x7f0a380fdc30 msgr2=0x7f0a380fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:13.876 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.466+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/1168807195 shutdown_connections 2023-07-12T15:23:13.876 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:13.466+0000 7f0a3ea0e700 1 -- 172.21.15.69:0/1168807195 wait complete. 2023-07-12T15:23:13.877 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1168807195' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-12T15:23:13.877 INFO:teuthology.orchestra.run.smithi069.stdout:Enabling cephadm module... 2023-07-12T15:23:14.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3085195476' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-12T15:23:14.948 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:14 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ignoring --setuser ceph since I am not root 2023-07-12T15:23:14.949 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:14 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ignoring --setgroup ceph since I am not root 2023-07-12T15:23:14.949 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:14 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:14.776+0000 7f5f5106d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-12T15:23:14.949 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:14 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:14.856+0000 7f5f5106d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-12T15:23:15.009 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.317+0000 7fc526c68700 1 Processor -- start 2023-07-12T15:23:15.009 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.317+0000 7fc526c68700 1 -- start start 2023-07-12T15:23:15.010 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.317+0000 7fc526c68700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 0x7fc5201043c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:15.010 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.317+0000 7fc526c68700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc520104900 con 0x7fc520103fb0 2023-07-12T15:23:15.010 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.317+0000 7fc524a04700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 0x7fc5201043c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:15.010 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.317+0000 7fc524a04700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 0x7fc5201043c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39584/0 (socket says 172.21.15.69:39584) 2023-07-12T15:23:15.010 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.317+0000 7fc524a04700 1 -- 172.21.15.69:0/3811459398 learned_addr learned my addr 172.21.15.69:0/3811459398 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:15.010 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.317+0000 7fc524a04700 1 -- 172.21.15.69:0/3811459398 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc520104a40 con 0x7fc520103fb0 2023-07-12T15:23:15.010 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.318+0000 7fc524a04700 1 --2- 172.21.15.69:0/3811459398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 0x7fc5201043c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc5140098a0 tx=0x7fc514009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0190b7c383653d server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:15.011 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.318+0000 7fc51f7fe700 1 -- 172.21.15.69:0/3811459398 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc514004160 con 0x7fc520103fb0 2023-07-12T15:23:15.011 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.318+0000 7fc51f7fe700 1 -- 172.21.15.69:0/3811459398 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7fc5140042c0 con 0x7fc520103fb0 2023-07-12T15:23:15.011 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.318+0000 7fc51f7fe700 1 -- 172.21.15.69:0/3811459398 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc514011900 con 0x7fc520103fb0 2023-07-12T15:23:15.011 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.318+0000 7fc526c68700 1 -- 172.21.15.69:0/3811459398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 msgr2=0x7fc5201043c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:15.011 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.318+0000 7fc526c68700 1 --2- 172.21.15.69:0/3811459398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 0x7fc5201043c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc5140098a0 tx=0x7fc514009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:15.011 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.318+0000 7fc526c68700 1 -- 172.21.15.69:0/3811459398 shutdown_connections 2023-07-12T15:23:15.012 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.318+0000 7fc526c68700 1 --2- 172.21.15.69:0/3811459398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 0x7fc5201043c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:15.012 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.318+0000 7fc526c68700 1 -- 172.21.15.69:0/3811459398 >> 172.21.15.69:0/3811459398 conn(0x7fc5200ff620 msgr2=0x7fc520101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:15.012 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.318+0000 7fc526c68700 1 -- 172.21.15.69:0/3811459398 shutdown_connections 2023-07-12T15:23:15.012 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.318+0000 7fc526c68700 1 -- 172.21.15.69:0/3811459398 wait complete. 2023-07-12T15:23:15.012 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.319+0000 7fc526c68700 1 Processor -- start 2023-07-12T15:23:15.012 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.319+0000 7fc526c68700 1 -- start start 2023-07-12T15:23:15.012 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.319+0000 7fc526c68700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 0x7fc520197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:15.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.319+0000 7fc526c68700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc520198190 con 0x7fc520103fb0 2023-07-12T15:23:15.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.319+0000 7fc524a04700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 0x7fc520197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:15.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.319+0000 7fc524a04700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 0x7fc520197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39586/0 (socket says 172.21.15.69:39586) 2023-07-12T15:23:15.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.319+0000 7fc524a04700 1 -- 172.21.15.69:0/3085195476 learned_addr learned my addr 172.21.15.69:0/3085195476 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:15.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.319+0000 7fc524a04700 1 -- 172.21.15.69:0/3085195476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc514009550 con 0x7fc520103fb0 2023-07-12T15:23:15.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.320+0000 7fc524a04700 1 --2- 172.21.15.69:0/3085195476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 0x7fc520197c50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc514006af0 tx=0x7fc514011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:15.013 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.320+0000 7fc51dffb700 1 -- 172.21.15.69:0/3085195476 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc514011970 con 0x7fc520103fb0 2023-07-12T15:23:15.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.320+0000 7fc51dffb700 1 -- 172.21.15.69:0/3085195476 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7fc514003b10 con 0x7fc520103fb0 2023-07-12T15:23:15.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.320+0000 7fc526c68700 1 -- 172.21.15.69:0/3085195476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc520198390 con 0x7fc520103fb0 2023-07-12T15:23:15.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.320+0000 7fc51dffb700 1 -- 172.21.15.69:0/3085195476 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc514024da0 con 0x7fc520103fb0 2023-07-12T15:23:15.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.320+0000 7fc526c68700 1 -- 172.21.15.69:0/3085195476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc520198830 con 0x7fc520103fb0 2023-07-12T15:23:15.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.321+0000 7fc51dffb700 1 -- 172.21.15.69:0/3085195476 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 4) v1 ==== 45019+0+0 (secure 0 0 0) 0x7fc5140036a0 con 0x7fc520103fb0 2023-07-12T15:23:15.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.321+0000 7fc526c68700 1 -- 172.21.15.69:0/3085195476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc520062130 con 0x7fc520103fb0 2023-07-12T15:23:15.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.321+0000 7fc51dffb700 1 --2- 172.21.15.69:0/3085195476 >> v2:172.21.15.69:6800/654231506 conn(0x7fc5100382e0 0x7fc51003a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:15.014 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.321+0000 7fc51dffb700 1 -- 172.21.15.69:0/3085195476 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc51404be40 con 0x7fc520103fb0 2023-07-12T15:23:15.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.321+0000 7fc51ffff700 1 --2- 172.21.15.69:0/3085195476 >> v2:172.21.15.69:6800/654231506 conn(0x7fc5100382e0 0x7fc51003a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:15.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.322+0000 7fc51ffff700 1 --2- 172.21.15.69:0/3085195476 >> v2:172.21.15.69:6800/654231506 conn(0x7fc5100382e0 0x7fc51003a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc50c006fd0 tx=0x7fc50c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:15.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.326+0000 7fc51dffb700 1 -- 172.21.15.69:0/3085195476 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc514003950 con 0x7fc520103fb0 2023-07-12T15:23:15.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.562+0000 7fc526c68700 1 -- 172.21.15.69:0/3085195476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fc52019b7b0 con 0x7fc520103fb0 2023-07-12T15:23:15.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.615+0000 7fc51dffb700 1 -- 172.21.15.69:0/3085195476 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 5) v1 ==== 45030+0+0 (secure 0 0 0) 0x7fc51402d910 con 0x7fc520103fb0 2023-07-12T15:23:15.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.616+0000 7fc51dffb700 1 -- 172.21.15.69:0/3085195476 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fc51404b860 con 0x7fc520103fb0 2023-07-12T15:23:15.015 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.619+0000 7fc526c68700 1 -- 172.21.15.69:0/3085195476 >> v2:172.21.15.69:6800/654231506 conn(0x7fc5100382e0 msgr2=0x7fc51003a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:15.016 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.619+0000 7fc526c68700 1 --2- 172.21.15.69:0/3085195476 >> v2:172.21.15.69:6800/654231506 conn(0x7fc5100382e0 0x7fc51003a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc50c006fd0 tx=0x7fc50c006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:15.016 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.619+0000 7fc526c68700 1 -- 172.21.15.69:0/3085195476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 msgr2=0x7fc520197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:15.016 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.619+0000 7fc526c68700 1 --2- 172.21.15.69:0/3085195476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 0x7fc520197c50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc514006af0 tx=0x7fc514011e80 comp rx=0 tx=0).stop 2023-07-12T15:23:15.016 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.619+0000 7fc526c68700 1 -- 172.21.15.69:0/3085195476 shutdown_connections 2023-07-12T15:23:15.017 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.619+0000 7fc526c68700 1 --2- 172.21.15.69:0/3085195476 >> v2:172.21.15.69:6800/654231506 conn(0x7fc5100382e0 0x7fc51003a790 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:15.017 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.619+0000 7fc526c68700 1 --2- 172.21.15.69:0/3085195476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc520103fb0 0x7fc520197c50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:15.017 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.619+0000 7fc526c68700 1 -- 172.21.15.69:0/3085195476 >> 172.21.15.69:0/3085195476 conn(0x7fc5200ff620 msgr2=0x7fc520100380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:15.017 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.619+0000 7fc526c68700 1 -- 172.21.15.69:0/3085195476 shutdown_connections 2023-07-12T15:23:15.017 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:14.619+0000 7fc526c68700 1 -- 172.21.15.69:0/3085195476 wait complete. 2023-07-12T15:23:15.220 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:14 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:14.947+0000 7f5f5106d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-12T15:23:15.220 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:15 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:15.046+0000 7f5f5106d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-12T15:23:15.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:15 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:15.218+0000 7f5f5106d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-12T15:23:15.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3085195476' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-12T15:23:15.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:15 smithi069 ceph-mon[94241]: mgrmap e5: y(active, since 11s) 2023-07-12T15:23:15.805 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:15 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:15.541+0000 7f5f5106d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-12T15:23:15.805 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:15 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:15.633+0000 7f5f5106d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-12T15:23:15.805 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:15 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:15.763+0000 7f5f5106d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-12T15:23:16.188 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:15 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:15.866+0000 7f5f5106d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-12T15:23:16.255 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2023-07-12T15:23:16.255 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-12T15:23:16.255 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-12T15:23:16.255 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "active_name": "y", 2023-07-12T15:23:16.256 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-12T15:23:16.256 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2023-07-12T15:23:16.256 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.514+0000 7ff57f971700 1 Processor -- start 2023-07-12T15:23:16.256 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.514+0000 7ff57f971700 1 -- start start 2023-07-12T15:23:16.256 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.514+0000 7ff57f971700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 0x7ff578103260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:16.256 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.514+0000 7ff57f971700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5781037a0 con 0x7ff578102e50 2023-07-12T15:23:16.256 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.515+0000 7ff57d70d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 0x7ff578103260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:16.257 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.515+0000 7ff57d70d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 0x7ff578103260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39600/0 (socket says 172.21.15.69:39600) 2023-07-12T15:23:16.257 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.515+0000 7ff57d70d700 1 -- 172.21.15.69:0/3512837293 learned_addr learned my addr 172.21.15.69:0/3512837293 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:16.257 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.515+0000 7ff57d70d700 1 -- 172.21.15.69:0/3512837293 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5781038e0 con 0x7ff578102e50 2023-07-12T15:23:16.257 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.515+0000 7ff57d70d700 1 --2- 172.21.15.69:0/3512837293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 0x7ff578103260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff57400bb40 tx=0x7ff57400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1ae60ca380ee4e server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:16.257 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.515+0000 7ff56ffff700 1 -- 172.21.15.69:0/3512837293 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff57400dbe0 con 0x7ff578102e50 2023-07-12T15:23:16.257 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.515+0000 7ff56ffff700 1 -- 172.21.15.69:0/3512837293 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7ff5740044a0 con 0x7ff578102e50 2023-07-12T15:23:16.257 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.515+0000 7ff56ffff700 1 -- 172.21.15.69:0/3512837293 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff574003d70 con 0x7ff578102e50 2023-07-12T15:23:16.258 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.516+0000 7ff57f971700 1 -- 172.21.15.69:0/3512837293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 msgr2=0x7ff578103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:16.258 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.516+0000 7ff57f971700 1 --2- 172.21.15.69:0/3512837293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 0x7ff578103260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff57400bb40 tx=0x7ff57400be50 comp rx=0 tx=0).stop 2023-07-12T15:23:16.258 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.516+0000 7ff57f971700 1 -- 172.21.15.69:0/3512837293 shutdown_connections 2023-07-12T15:23:16.258 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.516+0000 7ff57f971700 1 --2- 172.21.15.69:0/3512837293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 0x7ff578103260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:16.258 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.516+0000 7ff57f971700 1 -- 172.21.15.69:0/3512837293 >> 172.21.15.69:0/3512837293 conn(0x7ff5780fe410 msgr2=0x7ff578100880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:16.258 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.516+0000 7ff57f971700 1 -- 172.21.15.69:0/3512837293 shutdown_connections 2023-07-12T15:23:16.259 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.516+0000 7ff57f971700 1 -- 172.21.15.69:0/3512837293 wait complete. 2023-07-12T15:23:16.259 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.516+0000 7ff57f971700 1 Processor -- start 2023-07-12T15:23:16.259 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.516+0000 7ff57f971700 1 -- start start 2023-07-12T15:23:16.259 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.517+0000 7ff57f971700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 0x7ff578199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:16.259 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.517+0000 7ff57f971700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57819a3f0 con 0x7ff578102e50 2023-07-12T15:23:16.259 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.517+0000 7ff57d70d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 0x7ff578199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:16.259 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.517+0000 7ff57d70d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 0x7ff578199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39602/0 (socket says 172.21.15.69:39602) 2023-07-12T15:23:16.260 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.517+0000 7ff57d70d700 1 -- 172.21.15.69:0/2854386683 learned_addr learned my addr 172.21.15.69:0/2854386683 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:16.260 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.517+0000 7ff57d70d700 1 -- 172.21.15.69:0/2854386683 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57400b7f0 con 0x7ff578102e50 2023-07-12T15:23:16.261 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.517+0000 7ff57d70d700 1 --2- 172.21.15.69:0/2854386683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 0x7ff578199eb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff57400b340 tx=0x7ff5740041c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:16.261 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.517+0000 7ff56e7fc700 1 -- 172.21.15.69:0/2854386683 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5740106d0 con 0x7ff578102e50 2023-07-12T15:23:16.261 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.517+0000 7ff57f971700 1 -- 172.21.15.69:0/2854386683 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57819a5f0 con 0x7ff578102e50 2023-07-12T15:23:16.261 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.517+0000 7ff56e7fc700 1 -- 172.21.15.69:0/2854386683 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7ff574010ca0 con 0x7ff578102e50 2023-07-12T15:23:16.262 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.517+0000 7ff56e7fc700 1 -- 172.21.15.69:0/2854386683 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff574021c10 con 0x7ff578102e50 2023-07-12T15:23:16.262 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.517+0000 7ff57f971700 1 -- 172.21.15.69:0/2854386683 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57819aa30 con 0x7ff578102e50 2023-07-12T15:23:16.262 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.518+0000 7ff56e7fc700 1 -- 172.21.15.69:0/2854386683 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 45030+0+0 (secure 0 0 0) 0x7ff5740193f0 con 0x7ff578102e50 2023-07-12T15:23:16.262 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.518+0000 7ff57f971700 1 -- 172.21.15.69:0/2854386683 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57818f760 con 0x7ff578102e50 2023-07-12T15:23:16.263 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.518+0000 7ff56e7fc700 1 --2- 172.21.15.69:0/2854386683 >> v2:172.21.15.69:6800/654231506 conn(0x7ff564038330 0x7ff56403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:16.263 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.518+0000 7ff56e7fc700 1 -- 172.21.15.69:0/2854386683 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff57404c210 con 0x7ff578102e50 2023-07-12T15:23:16.263 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.518+0000 7ff57cf0c700 1 -- 172.21.15.69:0/2854386683 >> v2:172.21.15.69:6800/654231506 conn(0x7ff564038330 msgr2=0x7ff56403a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/654231506 2023-07-12T15:23:16.263 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.518+0000 7ff57cf0c700 1 --2- 172.21.15.69:0/2854386683 >> v2:172.21.15.69:6800/654231506 conn(0x7ff564038330 0x7ff56403a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-12T15:23:16.263 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.523+0000 7ff56e7fc700 1 -- 172.21.15.69:0/2854386683 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff57400e950 con 0x7ff578102e50 2023-07-12T15:23:16.264 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.719+0000 7ff57cf0c700 1 -- 172.21.15.69:0/2854386683 >> v2:172.21.15.69:6800/654231506 conn(0x7ff564038330 msgr2=0x7ff56403a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/654231506 2023-07-12T15:23:16.264 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.719+0000 7ff57cf0c700 1 --2- 172.21.15.69:0/2854386683 >> v2:172.21.15.69:6800/654231506 conn(0x7ff564038330 0x7ff56403a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-12T15:23:16.264 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.771+0000 7ff57f971700 1 -- 172.21.15.69:0/2854386683 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff57802cf40 con 0x7ff578102e50 2023-07-12T15:23:16.264 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.772+0000 7ff56e7fc700 1 -- 172.21.15.69:0/2854386683 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7ff57401d220 con 0x7ff578102e50 2023-07-12T15:23:16.265 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.774+0000 7ff57f971700 1 -- 172.21.15.69:0/2854386683 >> v2:172.21.15.69:6800/654231506 conn(0x7ff564038330 msgr2=0x7ff56403a7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-12T15:23:16.265 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.774+0000 7ff57f971700 1 --2- 172.21.15.69:0/2854386683 >> v2:172.21.15.69:6800/654231506 conn(0x7ff564038330 0x7ff56403a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:16.265 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.774+0000 7ff57f971700 1 -- 172.21.15.69:0/2854386683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 msgr2=0x7ff578199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:16.265 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.774+0000 7ff57f971700 1 --2- 172.21.15.69:0/2854386683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 0x7ff578199eb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff57400b340 tx=0x7ff5740041c0 comp rx=0 tx=0).stop 2023-07-12T15:23:16.266 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.774+0000 7ff57f971700 1 -- 172.21.15.69:0/2854386683 shutdown_connections 2023-07-12T15:23:16.266 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.774+0000 7ff57f971700 1 --2- 172.21.15.69:0/2854386683 >> v2:172.21.15.69:6800/654231506 conn(0x7ff564038330 0x7ff56403a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:16.266 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.774+0000 7ff57f971700 1 --2- 172.21.15.69:0/2854386683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff578102e50 0x7ff578199eb0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:16.266 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.774+0000 7ff57f971700 1 -- 172.21.15.69:0/2854386683 >> 172.21.15.69:0/2854386683 conn(0x7ff5780fe410 msgr2=0x7ff57818c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:16.267 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.774+0000 7ff57f971700 1 -- 172.21.15.69:0/2854386683 shutdown_connections 2023-07-12T15:23:16.267 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:15.774+0000 7ff57f971700 1 -- 172.21.15.69:0/2854386683 wait complete. 2023-07-12T15:23:16.267 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for the mgr to restart... 2023-07-12T15:23:16.267 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mgr epoch 5... 2023-07-12T15:23:16.474 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:16.303+0000 7f5f5106d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-12T15:23:16.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2854386683' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-12T15:23:17.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:16.881+0000 7f5f5106d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-12T15:23:17.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:17 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:17.047+0000 7f5f5106d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-12T15:23:18.053 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:17 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:17.793+0000 7f5f5106d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-12T15:23:18.054 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:17 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:17.883+0000 7f5f5106d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-12T15:23:18.330 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:18 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:18.052+0000 7f5f5106d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-12T15:23:18.700 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:18 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:18.328+0000 7f5f5106d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-12T15:23:18.701 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:18 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:18.553+0000 7f5f5106d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-12T15:23:18.961 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:18 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:18.699+0000 7f5f5106d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-12T15:23:19.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:18 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:18.960+0000 7f5f5106d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-12T15:23:19.969 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:19 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:19.652+0000 7f5f5106d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-12T15:23:19.970 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:19 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:19.780+0000 7f5f5106d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-12T15:23:20.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:19 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:19.968+0000 7f5f5106d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-12T15:23:20.236 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:20 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:20.079+0000 7f5f5106d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-12T15:23:20.735 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:20 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:20.291+0000 7f5f5106d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-12T15:23:21.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:21 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:21.129+0000 7f5f5106d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-12T15:23:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:21 smithi069 ceph-mon[94241]: Active manager daemon y restarted 2023-07-12T15:23:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:21 smithi069 ceph-mon[94241]: Activating manager daemon y 2023-07-12T15:23:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:21 smithi069 ceph-mon[94241]: osdmap e2: 0 total, 0 up, 0 in 2023-07-12T15:23:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:21 smithi069 ceph-mon[94241]: mgrmap e6: y(active, starting, since 0.00684812s) 2023-07-12T15:23:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:21 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:23:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:21 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-12T15:23:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:21 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-12T15:23:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:21 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-12T15:23:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:21 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-12T15:23:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:21 smithi069 ceph-mon[94241]: Manager daemon y is now available 2023-07-12T15:23:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:21 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:21 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:22.448 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:22 smithi069 ceph-mon[94241]: Found migration_current of "None". Setting to last migration. 2023-07-12T15:23:22.449 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:22 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:23:22.449 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:22 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:23:22.449 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:22 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:23:22.449 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:22 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-12T15:23:22.449 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:22 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-12T15:23:22.450 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:22 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:22.450 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:22 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:22.450 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:22 smithi069 ceph-mon[94241]: mgrmap e7: y(active, since 1.00489s) 2023-07-12T15:23:22.502 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2023-07-12T15:23:22.507 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-12T15:23:22.508 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-12T15:23:22.508 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2023-07-12T15:23:22.508 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.789+0000 7f0849c39700 1 Processor -- start 2023-07-12T15:23:22.508 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.789+0000 7f0849c39700 1 -- start start 2023-07-12T15:23:22.508 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.789+0000 7f0849c39700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 0x7f0844106df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:22.508 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.789+0000 7f0849c39700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0844077ec0 con 0x7f0844104a10 2023-07-12T15:23:22.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.789+0000 7f08437fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 0x7f0844106df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:22.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.789+0000 7f08437fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 0x7f0844106df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39610/0 (socket says 172.21.15.69:39610) 2023-07-12T15:23:22.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.789+0000 7f08437fe700 1 -- 172.21.15.69:0/167722168 learned_addr learned my addr 172.21.15.69:0/167722168 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:22.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.789+0000 7f08437fe700 1 -- 172.21.15.69:0/167722168 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0844107330 con 0x7f0844104a10 2023-07-12T15:23:22.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.790+0000 7f08437fe700 1 --2- 172.21.15.69:0/167722168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 0x7f0844106df0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f08340098a0 tx=0x7f0834009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a86e63632cdd2b6 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:22.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.790+0000 7f08427fc700 1 -- 172.21.15.69:0/167722168 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0834004160 con 0x7f0844104a10 2023-07-12T15:23:22.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.790+0000 7f08427fc700 1 -- 172.21.15.69:0/167722168 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f08340042c0 con 0x7f0844104a10 2023-07-12T15:23:22.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.790+0000 7f08427fc700 1 -- 172.21.15.69:0/167722168 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0834011900 con 0x7f0844104a10 2023-07-12T15:23:22.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.790+0000 7f0849c39700 1 -- 172.21.15.69:0/167722168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 msgr2=0x7f0844106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:22.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.790+0000 7f0849c39700 1 --2- 172.21.15.69:0/167722168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 0x7f0844106df0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f08340098a0 tx=0x7f0834009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:22.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.790+0000 7f0849c39700 1 -- 172.21.15.69:0/167722168 shutdown_connections 2023-07-12T15:23:22.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.790+0000 7f0849c39700 1 --2- 172.21.15.69:0/167722168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 0x7f0844106df0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:22.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.790+0000 7f0849c39700 1 -- 172.21.15.69:0/167722168 >> 172.21.15.69:0/167722168 conn(0x7f08440fe410 msgr2=0x7f0844100880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:22.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.790+0000 7f0849c39700 1 -- 172.21.15.69:0/167722168 shutdown_connections 2023-07-12T15:23:22.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.791+0000 7f0849c39700 1 -- 172.21.15.69:0/167722168 wait complete. 2023-07-12T15:23:22.511 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.791+0000 7f0849c39700 1 Processor -- start 2023-07-12T15:23:22.511 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.791+0000 7f0849c39700 1 -- start start 2023-07-12T15:23:22.511 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.791+0000 7f0849c39700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 0x7f0844197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:22.511 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.791+0000 7f0849c39700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0844198180 con 0x7f0844104a10 2023-07-12T15:23:22.511 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.792+0000 7f08437fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 0x7f0844197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:22.511 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.792+0000 7f08437fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 0x7f0844197c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39612/0 (socket says 172.21.15.69:39612) 2023-07-12T15:23:22.512 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.792+0000 7f08437fe700 1 -- 172.21.15.69:0/552456409 learned_addr learned my addr 172.21.15.69:0/552456409 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:22.512 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.792+0000 7f08437fe700 1 -- 172.21.15.69:0/552456409 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0834009550 con 0x7f0844104a10 2023-07-12T15:23:22.513 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.792+0000 7f08437fe700 1 --2- 172.21.15.69:0/552456409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 0x7f0844197c40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0834006af0 tx=0x7f0834011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:22.513 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.792+0000 7f0840ff9700 1 -- 172.21.15.69:0/552456409 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0834011d80 con 0x7f0844104a10 2023-07-12T15:23:22.513 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.792+0000 7f0840ff9700 1 -- 172.21.15.69:0/552456409 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f0834003b10 con 0x7f0844104a10 2023-07-12T15:23:22.513 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.792+0000 7f0840ff9700 1 -- 172.21.15.69:0/552456409 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f083401add0 con 0x7f0844104a10 2023-07-12T15:23:22.513 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.792+0000 7f0849c39700 1 -- 172.21.15.69:0/552456409 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0844198380 con 0x7f0844104a10 2023-07-12T15:23:22.513 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.792+0000 7f0849c39700 1 -- 172.21.15.69:0/552456409 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0844198820 con 0x7f0844104a10 2023-07-12T15:23:22.514 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.793+0000 7f0840ff9700 1 -- 172.21.15.69:0/552456409 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 45030+0+0 (secure 0 0 0) 0x7f083402d3f0 con 0x7f0844104a10 2023-07-12T15:23:22.514 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.793+0000 7f0840ff9700 1 --2- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 0x7f082c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:22.514 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.793+0000 7f0849c39700 1 -- 172.21.15.69:0/552456409 --> v2:172.21.15.69:6800/654231506 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08441080d0 con 0x7f082c0383d0 2023-07-12T15:23:22.514 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.793+0000 7f0840ff9700 1 -- 172.21.15.69:0/552456409 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f083404bfe0 con 0x7f0844104a10 2023-07-12T15:23:22.514 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.793+0000 7f0842ffd700 1 -- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 msgr2=0x7f082c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/654231506 2023-07-12T15:23:22.514 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.793+0000 7f0842ffd700 1 --2- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 0x7f082c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-12T15:23:22.514 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.994+0000 7f0842ffd700 1 -- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 msgr2=0x7f082c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/654231506 2023-07-12T15:23:22.515 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:16.994+0000 7f0842ffd700 1 --2- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 0x7f082c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-12T15:23:22.515 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:17.395+0000 7f0842ffd700 1 -- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 msgr2=0x7f082c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/654231506 2023-07-12T15:23:22.515 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:17.395+0000 7f0842ffd700 1 --2- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 0x7f082c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-12T15:23:22.515 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:18.196+0000 7f0842ffd700 1 -- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 msgr2=0x7f082c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/654231506 2023-07-12T15:23:22.515 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:18.196+0000 7f0842ffd700 1 --2- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 0x7f082c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-12T15:23:22.515 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:19.797+0000 7f0842ffd700 1 -- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 msgr2=0x7f082c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/654231506 2023-07-12T15:23:22.515 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:19.797+0000 7f0842ffd700 1 --2- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 0x7f082c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-12T15:23:22.516 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:21.148+0000 7f0840ff9700 1 -- 172.21.15.69:0/552456409 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mgrmap(e 6) v1 ==== 44832+0+0 (secure 0 0 0) 0x7f083402d880 con 0x7f0844104a10 2023-07-12T15:23:22.516 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:21.148+0000 7f0840ff9700 1 -- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 msgr2=0x7f082c03a880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-12T15:23:22.516 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:21.148+0000 7f0840ff9700 1 --2- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 0x7f082c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:22.516 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.145+0000 7f0840ff9700 1 -- 172.21.15.69:0/552456409 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 7) v1 ==== 44924+0+0 (secure 0 0 0) 0x7f083404d850 con 0x7f0844104a10 2023-07-12T15:23:22.516 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.145+0000 7f0840ff9700 1 --2- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/339337620 conn(0x7f082c03bab0 0x7f082c03de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:22.516 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.145+0000 7f0840ff9700 1 -- 172.21.15.69:0/552456409 --> v2:172.21.15.69:6800/339337620 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08441080d0 con 0x7f082c03bab0 2023-07-12T15:23:22.517 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.145+0000 7f0842ffd700 1 --2- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/339337620 conn(0x7f082c03bab0 0x7f082c03de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:22.517 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.146+0000 7f0842ffd700 1 --2- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/339337620 conn(0x7f082c03bab0 0x7f082c03de90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0838003a10 tx=0x7f08380092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:22.517 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.147+0000 7f0840ff9700 1 -- 172.21.15.69:0/552456409 <== mgr.14120 v2:172.21.15.69:6800/339337620 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f08441080d0 con 0x7f082c03bab0 2023-07-12T15:23:22.517 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.153+0000 7f0849c39700 1 -- 172.21.15.69:0/552456409 --> v2:172.21.15.69:6800/339337620 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0844108240 con 0x7f082c03bab0 2023-07-12T15:23:22.517 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.153+0000 7f0840ff9700 1 -- 172.21.15.69:0/552456409 <== mgr.14120 v2:172.21.15.69:6800/339337620 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f0844108240 con 0x7f082c03bab0 2023-07-12T15:23:22.518 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.153+0000 7f0849c39700 1 -- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/339337620 conn(0x7f082c03bab0 msgr2=0x7f082c03de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:22.518 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.153+0000 7f0849c39700 1 --2- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/339337620 conn(0x7f082c03bab0 0x7f082c03de90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0838003a10 tx=0x7f08380092b0 comp rx=0 tx=0).stop 2023-07-12T15:23:22.518 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.153+0000 7f0849c39700 1 -- 172.21.15.69:0/552456409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 msgr2=0x7f0844197c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:22.518 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.153+0000 7f0849c39700 1 --2- 172.21.15.69:0/552456409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 0x7f0844197c40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0834006af0 tx=0x7f0834011980 comp rx=0 tx=0).stop 2023-07-12T15:23:22.518 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.153+0000 7f0849c39700 1 -- 172.21.15.69:0/552456409 shutdown_connections 2023-07-12T15:23:22.518 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.153+0000 7f0849c39700 1 --2- 172.21.15.69:0/552456409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0844104a10 0x7f0844197c40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:22.518 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.154+0000 7f0849c39700 1 --2- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/339337620 conn(0x7f082c03bab0 0x7f082c03de90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:22.519 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.154+0000 7f0849c39700 1 --2- 172.21.15.69:0/552456409 >> v2:172.21.15.69:6800/654231506 conn(0x7f082c0383d0 0x7f082c03a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:22.519 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.154+0000 7f0849c39700 1 -- 172.21.15.69:0/552456409 >> 172.21.15.69:0/552456409 conn(0x7f08440fe410 msgr2=0x7f08440ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:22.519 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.154+0000 7f0849c39700 1 -- 172.21.15.69:0/552456409 shutdown_connections 2023-07-12T15:23:22.519 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.154+0000 7f0849c39700 1 -- 172.21.15.69:0/552456409 wait complete. 2023-07-12T15:23:22.519 INFO:teuthology.orchestra.run.smithi069.stdout:mgr epoch 5 is available 2023-07-12T15:23:22.519 INFO:teuthology.orchestra.run.smithi069.stdout:Setting orchestrator backend to cephadm... 2023-07-12T15:23:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:23 smithi069 ceph-mon[94241]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-12T15:23:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:23 smithi069 ceph-mon[94241]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-12T15:23:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:23 smithi069 ceph-mon[94241]: [12/Jul/2023:15:23:22] ENGINE Bus STARTING 2023-07-12T15:23:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:23 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:23:23.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:23 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:23.661 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.980+0000 7f04b4903700 1 Processor -- start 2023-07-12T15:23:23.662 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.980+0000 7f04b4903700 1 -- start start 2023-07-12T15:23:23.662 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.980+0000 7f04b4903700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 0x7f04b01043c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:23.662 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.980+0000 7f04b4903700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0104900 con 0x7f04b0103fb0 2023-07-12T15:23:23.662 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.981+0000 7f04ae59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 0x7f04b01043c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:23.662 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.981+0000 7f04ae59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 0x7f04b01043c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39664/0 (socket says 172.21.15.69:39664) 2023-07-12T15:23:23.663 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.981+0000 7f04ae59c700 1 -- 172.21.15.69:0/1382073925 learned_addr learned my addr 172.21.15.69:0/1382073925 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:23.663 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.981+0000 7f04ae59c700 1 -- 172.21.15.69:0/1382073925 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b0104a40 con 0x7f04b0103fb0 2023-07-12T15:23:23.663 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.981+0000 7f04ae59c700 1 --2- 172.21.15.69:0/1382073925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 0x7f04b01043c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f04a40098a0 tx=0x7f04a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1598171b11cc258 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:23.663 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.981+0000 7f04ad59a700 1 -- 172.21.15.69:0/1382073925 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04a4004160 con 0x7f04b0103fb0 2023-07-12T15:23:23.663 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.981+0000 7f04ad59a700 1 -- 172.21.15.69:0/1382073925 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f04a40042c0 con 0x7f04b0103fb0 2023-07-12T15:23:23.664 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.981+0000 7f04ad59a700 1 -- 172.21.15.69:0/1382073925 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04a4011900 con 0x7f04b0103fb0 2023-07-12T15:23:23.665 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.982+0000 7f04b4903700 1 -- 172.21.15.69:0/1382073925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 msgr2=0x7f04b01043c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:23.665 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.982+0000 7f04b4903700 1 --2- 172.21.15.69:0/1382073925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 0x7f04b01043c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f04a40098a0 tx=0x7f04a4009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:23.665 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.982+0000 7f04b4903700 1 -- 172.21.15.69:0/1382073925 shutdown_connections 2023-07-12T15:23:23.665 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.982+0000 7f04b4903700 1 --2- 172.21.15.69:0/1382073925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 0x7f04b01043c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:23.665 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.982+0000 7f04b4903700 1 -- 172.21.15.69:0/1382073925 >> 172.21.15.69:0/1382073925 conn(0x7f04b00ff620 msgr2=0x7f04b0101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:23.665 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.982+0000 7f04b4903700 1 -- 172.21.15.69:0/1382073925 shutdown_connections 2023-07-12T15:23:23.665 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.982+0000 7f04b4903700 1 -- 172.21.15.69:0/1382073925 wait complete. 2023-07-12T15:23:23.666 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f04b4903700 1 Processor -- start 2023-07-12T15:23:23.666 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f04b4903700 1 -- start start 2023-07-12T15:23:23.666 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f04b4903700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 0x7f04b0197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:23.666 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f04b4903700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0198190 con 0x7f04b0103fb0 2023-07-12T15:23:23.666 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f04ae59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 0x7f04b0197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:23.666 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f04ae59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 0x7f04b0197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39666/0 (socket says 172.21.15.69:39666) 2023-07-12T15:23:23.666 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f04ae59c700 1 -- 172.21.15.69:0/2378857700 learned_addr learned my addr 172.21.15.69:0/2378857700 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:23.667 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f04ae59c700 1 -- 172.21.15.69:0/2378857700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a4009550 con 0x7f04b0103fb0 2023-07-12T15:23:23.667 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f04ae59c700 1 --2- 172.21.15.69:0/2378857700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 0x7f04b0197c50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f04a4006af0 tx=0x7f04a4011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:23.667 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f049f7fe700 1 -- 172.21.15.69:0/2378857700 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04a4011d80 con 0x7f04b0103fb0 2023-07-12T15:23:23.667 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f049f7fe700 1 -- 172.21.15.69:0/2378857700 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f04a4003b10 con 0x7f04b0103fb0 2023-07-12T15:23:23.667 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f04b4903700 1 -- 172.21.15.69:0/2378857700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b0198390 con 0x7f04b0103fb0 2023-07-12T15:23:23.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f049f7fe700 1 -- 172.21.15.69:0/2378857700 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04a401adb0 con 0x7f04b0103fb0 2023-07-12T15:23:23.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.983+0000 7f04b4903700 1 -- 172.21.15.69:0/2378857700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b0198830 con 0x7f04b0103fb0 2023-07-12T15:23:23.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.984+0000 7f049f7fe700 1 -- 172.21.15.69:0/2378857700 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 7) v1 ==== 44924+0+0 (secure 0 0 0) 0x7f04a4024440 con 0x7f04b0103fb0 2023-07-12T15:23:23.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.984+0000 7f04b4903700 1 -- 172.21.15.69:0/2378857700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b0062130 con 0x7f04b0103fb0 2023-07-12T15:23:23.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.984+0000 7f049f7fe700 1 --2- 172.21.15.69:0/2378857700 >> v2:172.21.15.69:6800/339337620 conn(0x7f0498038260 0x7f049803a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:23.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.984+0000 7f049f7fe700 1 -- 172.21.15.69:0/2378857700 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f04a404b950 con 0x7f04b0103fb0 2023-07-12T15:23:23.669 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.985+0000 7f04add9b700 1 --2- 172.21.15.69:0/2378857700 >> v2:172.21.15.69:6800/339337620 conn(0x7f0498038260 0x7f049803a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:23.669 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.985+0000 7f04add9b700 1 --2- 172.21.15.69:0/2378857700 >> v2:172.21.15.69:6800/339337620 conn(0x7f0498038260 0x7f049803a710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f04a0006fd0 tx=0x7f04a0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:23.669 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:22.989+0000 7f049f7fe700 1 -- 172.21.15.69:0/2378857700 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f04a40246f0 con 0x7f04b0103fb0 2023-07-12T15:23:23.669 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:23.188+0000 7f04b4903700 1 -- 172.21.15.69:0/2378857700 --> v2:172.21.15.69:6800/339337620 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f04b0100a90 con 0x7f0498038260 2023-07-12T15:23:23.669 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:23.194+0000 7f049f7fe700 1 -- 172.21.15.69:0/2378857700 <== mgr.14120 v2:172.21.15.69:6800/339337620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f04b0100a90 con 0x7f0498038260 2023-07-12T15:23:23.669 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:23.196+0000 7f04b4903700 1 -- 172.21.15.69:0/2378857700 >> v2:172.21.15.69:6800/339337620 conn(0x7f0498038260 msgr2=0x7f049803a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:23.669 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:23.196+0000 7f04b4903700 1 --2- 172.21.15.69:0/2378857700 >> v2:172.21.15.69:6800/339337620 conn(0x7f0498038260 0x7f049803a710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f04a0006fd0 tx=0x7f04a0006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:23.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:23.196+0000 7f04b4903700 1 -- 172.21.15.69:0/2378857700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 msgr2=0x7f04b0197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:23.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:23.196+0000 7f04b4903700 1 --2- 172.21.15.69:0/2378857700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 0x7f04b0197c50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f04a4006af0 tx=0x7f04a4011980 comp rx=0 tx=0).stop 2023-07-12T15:23:23.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:23.197+0000 7f04b4903700 1 -- 172.21.15.69:0/2378857700 shutdown_connections 2023-07-12T15:23:23.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:23.197+0000 7f04b4903700 1 --2- 172.21.15.69:0/2378857700 >> v2:172.21.15.69:6800/339337620 conn(0x7f0498038260 0x7f049803a710 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:23.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:23.197+0000 7f04b4903700 1 --2- 172.21.15.69:0/2378857700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b0103fb0 0x7f04b0197c50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:23.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:23.197+0000 7f04b4903700 1 -- 172.21.15.69:0/2378857700 >> 172.21.15.69:0/2378857700 conn(0x7f04b00ff620 msgr2=0x7f04b0100380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:23.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:23.197+0000 7f04b4903700 1 -- 172.21.15.69:0/2378857700 shutdown_connections 2023-07-12T15:23:23.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:23.197+0000 7f04b4903700 1 -- 172.21.15.69:0/2378857700 wait complete. 2023-07-12T15:23:24.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:24 smithi069 ceph-mon[94241]: [12/Jul/2023:15:23:22] ENGINE Serving on http://172.21.15.69:8765 2023-07-12T15:23:24.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:24 smithi069 ceph-mon[94241]: [12/Jul/2023:15:23:22] ENGINE Serving on https://172.21.15.69:7150 2023-07-12T15:23:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:24 smithi069 ceph-mon[94241]: [12/Jul/2023:15:23:22] ENGINE Bus STARTED 2023-07-12T15:23:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:24 smithi069 ceph-mon[94241]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:23:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:24 smithi069 ceph-mon[94241]: mgrmap e8: y(active, since 2s) 2023-07-12T15:23:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:24 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:23:24.809 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-12T15:23:24.809 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.177+0000 7f92c5598700 1 Processor -- start 2023-07-12T15:23:24.809 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.177+0000 7f92c5598700 1 -- start start 2023-07-12T15:23:24.809 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.178+0000 7f92c5598700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 0x7f92c0107070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:24.810 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.178+0000 7f92c5598700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c0077e60 con 0x7f92c0104c90 2023-07-12T15:23:24.810 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.178+0000 7f92beffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 0x7f92c0107070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:24.810 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.178+0000 7f92beffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 0x7f92c0107070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39670/0 (socket says 172.21.15.69:39670) 2023-07-12T15:23:24.810 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.178+0000 7f92beffd700 1 -- 172.21.15.69:0/3035567312 learned_addr learned my addr 172.21.15.69:0/3035567312 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:24.810 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.178+0000 7f92beffd700 1 -- 172.21.15.69:0/3035567312 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c01075b0 con 0x7f92c0104c90 2023-07-12T15:23:24.810 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.178+0000 7f92beffd700 1 --2- 172.21.15.69:0/3035567312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 0x7f92c0107070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f92b40098a0 tx=0x7f92b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d9a04aebf10e99 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:24.810 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.178+0000 7f92bdffb700 1 -- 172.21.15.69:0/3035567312 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92b4004160 con 0x7f92c0104c90 2023-07-12T15:23:24.811 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.178+0000 7f92bdffb700 1 -- 172.21.15.69:0/3035567312 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f92b40042c0 con 0x7f92c0104c90 2023-07-12T15:23:24.811 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.178+0000 7f92bdffb700 1 -- 172.21.15.69:0/3035567312 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92b4011900 con 0x7f92c0104c90 2023-07-12T15:23:24.811 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.179+0000 7f92c5598700 1 -- 172.21.15.69:0/3035567312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 msgr2=0x7f92c0107070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:24.811 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.179+0000 7f92c5598700 1 --2- 172.21.15.69:0/3035567312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 0x7f92c0107070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f92b40098a0 tx=0x7f92b4009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:24.811 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.179+0000 7f92c5598700 1 -- 172.21.15.69:0/3035567312 shutdown_connections 2023-07-12T15:23:24.811 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.179+0000 7f92c5598700 1 --2- 172.21.15.69:0/3035567312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 0x7f92c0107070 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:24.811 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.179+0000 7f92c5598700 1 -- 172.21.15.69:0/3035567312 >> 172.21.15.69:0/3035567312 conn(0x7f92c00fe630 msgr2=0x7f92c0100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:24.812 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.179+0000 7f92c5598700 1 -- 172.21.15.69:0/3035567312 shutdown_connections 2023-07-12T15:23:24.812 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.179+0000 7f92c5598700 1 -- 172.21.15.69:0/3035567312 wait complete. 2023-07-12T15:23:24.812 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.179+0000 7f92c5598700 1 Processor -- start 2023-07-12T15:23:24.812 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.180+0000 7f92c5598700 1 -- start start 2023-07-12T15:23:24.812 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.180+0000 7f92c5598700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 0x7f92c0197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:24.812 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.180+0000 7f92c5598700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c0198320 con 0x7f92c0104c90 2023-07-12T15:23:24.812 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.180+0000 7f92beffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 0x7f92c0197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:24.813 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.180+0000 7f92beffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 0x7f92c0197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39672/0 (socket says 172.21.15.69:39672) 2023-07-12T15:23:24.813 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.180+0000 7f92beffd700 1 -- 172.21.15.69:0/2865136249 learned_addr learned my addr 172.21.15.69:0/2865136249 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:24.813 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.180+0000 7f92beffd700 1 -- 172.21.15.69:0/2865136249 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b4009550 con 0x7f92c0104c90 2023-07-12T15:23:24.813 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.180+0000 7f92beffd700 1 --2- 172.21.15.69:0/2865136249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 0x7f92c0197de0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f92b4006af0 tx=0x7f92b4011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:24.813 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.180+0000 7f92a7fff700 1 -- 172.21.15.69:0/2865136249 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92b4011d80 con 0x7f92c0104c90 2023-07-12T15:23:24.813 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.180+0000 7f92a7fff700 1 -- 172.21.15.69:0/2865136249 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f92b4003b10 con 0x7f92c0104c90 2023-07-12T15:23:24.813 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.180+0000 7f92c5598700 1 -- 172.21.15.69:0/2865136249 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c0198520 con 0x7f92c0104c90 2023-07-12T15:23:24.813 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.181+0000 7f92a7fff700 1 -- 172.21.15.69:0/2865136249 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92b401adb0 con 0x7f92c0104c90 2023-07-12T15:23:24.814 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.181+0000 7f92c5598700 1 -- 172.21.15.69:0/2865136249 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c01989c0 con 0x7f92c0104c90 2023-07-12T15:23:24.814 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.181+0000 7f92a7fff700 1 -- 172.21.15.69:0/2865136249 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) v1 ==== 45030+0+0 (secure 0 0 0) 0x7f92b4024440 con 0x7f92c0104c90 2023-07-12T15:23:24.814 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.181+0000 7f92c5598700 1 -- 172.21.15.69:0/2865136249 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c0062130 con 0x7f92c0104c90 2023-07-12T15:23:24.814 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.181+0000 7f92a7fff700 1 --2- 172.21.15.69:0/2865136249 >> v2:172.21.15.69:6800/339337620 conn(0x7f92a8038380 0x7f92a803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:24.814 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.182+0000 7f92a7fff700 1 -- 172.21.15.69:0/2865136249 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f92b404bb30 con 0x7f92c0104c90 2023-07-12T15:23:24.814 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.182+0000 7f92be7fc700 1 --2- 172.21.15.69:0/2865136249 >> v2:172.21.15.69:6800/339337620 conn(0x7f92a8038380 0x7f92a803a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:24.814 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.182+0000 7f92be7fc700 1 --2- 172.21.15.69:0/2865136249 >> v2:172.21.15.69:6800/339337620 conn(0x7f92a8038380 0x7f92a803a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f92b0006fd0 tx=0x7f92b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:24.815 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.186+0000 7f92a7fff700 1 -- 172.21.15.69:0/2865136249 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f92b40246f0 con 0x7f92c0104c90 2023-07-12T15:23:24.815 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.374+0000 7f92c5598700 1 -- 172.21.15.69:0/2865136249 --> v2:172.21.15.69:6800/339337620 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f92c00ffb10 con 0x7f92a8038380 2023-07-12T15:23:24.815 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.374+0000 7f92a7fff700 1 -- 172.21.15.69:0/2865136249 <== mgr.14120 v2:172.21.15.69:6800/339337620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f92c00ffb10 con 0x7f92a8038380 2023-07-12T15:23:24.815 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.376+0000 7f92c5598700 1 -- 172.21.15.69:0/2865136249 >> v2:172.21.15.69:6800/339337620 conn(0x7f92a8038380 msgr2=0x7f92a803a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:24.815 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.376+0000 7f92c5598700 1 --2- 172.21.15.69:0/2865136249 >> v2:172.21.15.69:6800/339337620 conn(0x7f92a8038380 0x7f92a803a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f92b0006fd0 tx=0x7f92b0006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:24.815 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.376+0000 7f92c5598700 1 -- 172.21.15.69:0/2865136249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 msgr2=0x7f92c0197de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:24.815 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.376+0000 7f92c5598700 1 --2- 172.21.15.69:0/2865136249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 0x7f92c0197de0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f92b4006af0 tx=0x7f92b4011980 comp rx=0 tx=0).stop 2023-07-12T15:23:24.816 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.376+0000 7f92c5598700 1 -- 172.21.15.69:0/2865136249 shutdown_connections 2023-07-12T15:23:24.816 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.376+0000 7f92c5598700 1 --2- 172.21.15.69:0/2865136249 >> v2:172.21.15.69:6800/339337620 conn(0x7f92a8038380 0x7f92a803a830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:24.816 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.376+0000 7f92c5598700 1 --2- 172.21.15.69:0/2865136249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c0104c90 0x7f92c0197de0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:24.816 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.376+0000 7f92c5598700 1 -- 172.21.15.69:0/2865136249 >> 172.21.15.69:0/2865136249 conn(0x7f92c00fe630 msgr2=0x7f92c00ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:24.816 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.377+0000 7f92c5598700 1 -- 172.21.15.69:0/2865136249 shutdown_connections 2023-07-12T15:23:24.816 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:24.377+0000 7f92c5598700 1 -- 172.21.15.69:0/2865136249 wait complete. 2023-07-12T15:23:24.816 INFO:teuthology.orchestra.run.smithi069.stdout:Generating ssh key... 2023-07-12T15:23:25.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-mon[94241]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:23:26.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:26 smithi069 ceph-mon[94241]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:23:26.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:26 smithi069 ceph-mon[94241]: Generating ssh key... 2023-07-12T15:23:26.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:26 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:26.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:26 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:26.395 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: Generating public/private rsa key pair. 2023-07-12T15:23:26.395 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: Your identification has been saved in /tmp/tmph19s4_yn/key. 2023-07-12T15:23:26.396 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: Your public key has been saved in /tmp/tmph19s4_yn/key.pub. 2023-07-12T15:23:26.396 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: The key fingerprint is: 2023-07-12T15:23:26.396 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: SHA256:tY3J0sDYgjrHllE0GCT2Mc+RUSqp19NneWtBz8EHeEI ceph-dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:23:26.396 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: The key's randomart image is: 2023-07-12T15:23:26.396 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: +---[RSA 3072]----+ 2023-07-12T15:23:26.397 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: | o.=+*+. .E.. | 2023-07-12T15:23:26.397 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: | . o.O.B o... | 2023-07-12T15:23:26.397 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: | * * + . .oo .| 2023-07-12T15:23:26.397 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: | + = o = B o o | 2023-07-12T15:23:26.397 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: | + * o S X + o | 2023-07-12T15:23:26.397 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: | = . + . o | 2023-07-12T15:23:26.398 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: | o | 2023-07-12T15:23:26.398 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: | . | 2023-07-12T15:23:26.398 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: | | 2023-07-12T15:23:26.398 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:25 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: +----[SHA256]-----+ 2023-07-12T15:23:26.453 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.273+0000 7f170e800700 1 Processor -- start 2023-07-12T15:23:26.454 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.273+0000 7f170e800700 1 -- start start 2023-07-12T15:23:26.454 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.273+0000 7f170e800700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 0x7f1708103260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:26.454 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.273+0000 7f170e800700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17081037a0 con 0x7f1708102e50 2023-07-12T15:23:26.454 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.273+0000 7f1707fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 0x7f1708103260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:26.454 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.273+0000 7f1707fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 0x7f1708103260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39676/0 (socket says 172.21.15.69:39676) 2023-07-12T15:23:26.455 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.273+0000 7f1707fff700 1 -- 172.21.15.69:0/1674605276 learned_addr learned my addr 172.21.15.69:0/1674605276 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:26.455 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.273+0000 7f1707fff700 1 -- 172.21.15.69:0/1674605276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17081038e0 con 0x7f1708102e50 2023-07-12T15:23:26.455 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.273+0000 7f1707fff700 1 --2- 172.21.15.69:0/1674605276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 0x7f1708103260 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f16f80098a0 tx=0x7f16f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15549e9a77d4135 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:26.455 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.274+0000 7f1706ffd700 1 -- 172.21.15.69:0/1674605276 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16f8004160 con 0x7f1708102e50 2023-07-12T15:23:26.455 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.274+0000 7f1706ffd700 1 -- 172.21.15.69:0/1674605276 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f16f80042c0 con 0x7f1708102e50 2023-07-12T15:23:26.456 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.274+0000 7f1706ffd700 1 -- 172.21.15.69:0/1674605276 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16f8011900 con 0x7f1708102e50 2023-07-12T15:23:26.456 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.274+0000 7f170e800700 1 -- 172.21.15.69:0/1674605276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 msgr2=0x7f1708103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:26.456 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.274+0000 7f170e800700 1 --2- 172.21.15.69:0/1674605276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 0x7f1708103260 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f16f80098a0 tx=0x7f16f8009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:26.456 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.274+0000 7f170e800700 1 -- 172.21.15.69:0/1674605276 shutdown_connections 2023-07-12T15:23:26.456 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.274+0000 7f170e800700 1 --2- 172.21.15.69:0/1674605276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 0x7f1708103260 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:26.457 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.274+0000 7f170e800700 1 -- 172.21.15.69:0/1674605276 >> 172.21.15.69:0/1674605276 conn(0x7f17080fe410 msgr2=0x7f1708100880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:26.457 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.274+0000 7f170e800700 1 -- 172.21.15.69:0/1674605276 shutdown_connections 2023-07-12T15:23:26.458 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.274+0000 7f170e800700 1 -- 172.21.15.69:0/1674605276 wait complete. 2023-07-12T15:23:26.458 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.275+0000 7f170e800700 1 Processor -- start 2023-07-12T15:23:26.458 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.275+0000 7f170e800700 1 -- start start 2023-07-12T15:23:26.459 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.275+0000 7f170e800700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 0x7f1708197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:26.459 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.275+0000 7f170e800700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17081981e0 con 0x7f1708102e50 2023-07-12T15:23:26.459 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.275+0000 7f1707fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 0x7f1708197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:26.459 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.275+0000 7f1707fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 0x7f1708197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39678/0 (socket says 172.21.15.69:39678) 2023-07-12T15:23:26.459 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.275+0000 7f1707fff700 1 -- 172.21.15.69:0/3838418094 learned_addr learned my addr 172.21.15.69:0/3838418094 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:26.459 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.276+0000 7f1707fff700 1 -- 172.21.15.69:0/3838418094 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f8009550 con 0x7f1708102e50 2023-07-12T15:23:26.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.276+0000 7f1707fff700 1 --2- 172.21.15.69:0/3838418094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 0x7f1708197ca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f16f8006af0 tx=0x7f16f8011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:26.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.276+0000 7f17057fa700 1 -- 172.21.15.69:0/3838418094 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16f8011e40 con 0x7f1708102e50 2023-07-12T15:23:26.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.276+0000 7f17057fa700 1 -- 172.21.15.69:0/3838418094 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f16f8003b10 con 0x7f1708102e50 2023-07-12T15:23:26.460 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.276+0000 7f170e800700 1 -- 172.21.15.69:0/3838418094 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17081983e0 con 0x7f1708102e50 2023-07-12T15:23:26.461 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.276+0000 7f17057fa700 1 -- 172.21.15.69:0/3838418094 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16f8024e20 con 0x7f1708102e50 2023-07-12T15:23:26.461 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.276+0000 7f170e800700 1 -- 172.21.15.69:0/3838418094 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1708198880 con 0x7f1708102e50 2023-07-12T15:23:26.461 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.277+0000 7f17057fa700 1 -- 172.21.15.69:0/3838418094 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) v1 ==== 45030+0+0 (secure 0 0 0) 0x7f16f801a450 con 0x7f1708102e50 2023-07-12T15:23:26.461 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.277+0000 7f17057fa700 1 --2- 172.21.15.69:0/3838418094 >> v2:172.21.15.69:6800/339337620 conn(0x7f16f0038380 0x7f16f003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:26.462 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.277+0000 7f170e800700 1 -- 172.21.15.69:0/3838418094 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1708062130 con 0x7f1708102e50 2023-07-12T15:23:26.462 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.277+0000 7f17057fa700 1 -- 172.21.15.69:0/3838418094 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f16f804bd90 con 0x7f1708102e50 2023-07-12T15:23:26.462 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.277+0000 7f17077fe700 1 --2- 172.21.15.69:0/3838418094 >> v2:172.21.15.69:6800/339337620 conn(0x7f16f0038380 0x7f16f003a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:26.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.277+0000 7f17077fe700 1 --2- 172.21.15.69:0/3838418094 >> v2:172.21.15.69:6800/339337620 conn(0x7f16f0038380 0x7f16f003a830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f16fc006fd0 tx=0x7f16fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:26.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.282+0000 7f17057fa700 1 -- 172.21.15.69:0/3838418094 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f16f801a700 con 0x7f1708102e50 2023-07-12T15:23:26.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.467+0000 7f170e800700 1 -- 172.21.15.69:0/3838418094 --> v2:172.21.15.69:6800/339337620 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f1708100770 con 0x7f16f0038380 2023-07-12T15:23:26.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.993+0000 7f17057fa700 1 -- 172.21.15.69:0/3838418094 <== mgr.14120 v2:172.21.15.69:6800/339337620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1708100770 con 0x7f16f0038380 2023-07-12T15:23:26.463 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.995+0000 7f170e800700 1 -- 172.21.15.69:0/3838418094 >> v2:172.21.15.69:6800/339337620 conn(0x7f16f0038380 msgr2=0x7f16f003a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:26.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.995+0000 7f170e800700 1 --2- 172.21.15.69:0/3838418094 >> v2:172.21.15.69:6800/339337620 conn(0x7f16f0038380 0x7f16f003a830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f16fc006fd0 tx=0x7f16fc006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:26.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.996+0000 7f170e800700 1 -- 172.21.15.69:0/3838418094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 msgr2=0x7f1708197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:26.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.996+0000 7f170e800700 1 --2- 172.21.15.69:0/3838418094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 0x7f1708197ca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f16f8006af0 tx=0x7f16f8011980 comp rx=0 tx=0).stop 2023-07-12T15:23:26.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.996+0000 7f170e800700 1 -- 172.21.15.69:0/3838418094 shutdown_connections 2023-07-12T15:23:26.464 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.996+0000 7f170e800700 1 --2- 172.21.15.69:0/3838418094 >> v2:172.21.15.69:6800/339337620 conn(0x7f16f0038380 0x7f16f003a830 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:26.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.996+0000 7f170e800700 1 --2- 172.21.15.69:0/3838418094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1708102e50 0x7f1708197ca0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:26.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.996+0000 7f170e800700 1 -- 172.21.15.69:0/3838418094 >> 172.21.15.69:0/3838418094 conn(0x7f17080fe410 msgr2=0x7f17080ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:26.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.996+0000 7f170e800700 1 -- 172.21.15.69:0/3838418094 shutdown_connections 2023-07-12T15:23:26.465 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:25.996+0000 7f170e800700 1 -- 172.21.15.69:0/3838418094 wait complete. 2023-07-12T15:23:27.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx5AZBZo8AuiArArcXOFtG4+RnBzbgzZdifySyikikFatg/V+eSMp0CnjS7FiTJ78OlRtDWHqyQEuwhAqNfn/q9pznEvgP2XXiS6aaOuSuXmj3mvt0v2ROtVcs/z0dFu5QPD0wxPcJ9ZmFDWCipLqT+cd7muGV3/VDx8VYtrRFoiu2pyFlnVkV15HkkUN9yWT4QjusKWIUjEmVdHzIz4yzsSNR2XOCu8+WRP+pCTs+Z+10DtXVzBb5qHYCYHEAlPkfg2P88BxbkPC9JBAj5A7iFjjYIe5qLA7a2z2xX/gCZxZgvv6S6FwssNVXiZfkc+b2kWGXDY9A4eRU8uKfAoc4QYDwoJStH069gnbuJlYuxKNugm5URUA8WnUbYVcNhTXaN2jdulG9Jvhm3vQMI5o4BeTcey8sVaWgasEHPK/UfhqrfEriiABaBn2IIKItsZ8Tycm2cEQFfg2Ik6w4HHv9uScPBkoYGgeQybvHejM68B2Uqqyq3SQ602I8n8el0UM= ceph-dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:23:27.494 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.922+0000 7fab9780a700 1 Processor -- start 2023-07-12T15:23:27.495 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.922+0000 7fab9780a700 1 -- start start 2023-07-12T15:23:27.495 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.922+0000 7fab9780a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 0x7fab901026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:27.495 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.922+0000 7fab9780a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab90102c90 con 0x7fab901022b0 2023-07-12T15:23:27.495 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.923+0000 7fab955a6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 0x7fab901026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:27.496 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.923+0000 7fab955a6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 0x7fab901026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39682/0 (socket says 172.21.15.69:39682) 2023-07-12T15:23:27.496 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.923+0000 7fab955a6700 1 -- 172.21.15.69:0/3806876688 learned_addr learned my addr 172.21.15.69:0/3806876688 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:27.496 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.923+0000 7fab955a6700 1 -- 172.21.15.69:0/3806876688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab90105240 con 0x7fab901022b0 2023-07-12T15:23:27.496 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.923+0000 7fab955a6700 1 --2- 172.21.15.69:0/3806876688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 0x7fab901026c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fab800098a0 tx=0x7fab80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f643a795ce54021a server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:27.497 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.923+0000 7fab87fff700 1 -- 172.21.15.69:0/3806876688 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab80004160 con 0x7fab901022b0 2023-07-12T15:23:27.498 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.923+0000 7fab87fff700 1 -- 172.21.15.69:0/3806876688 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7fab800042c0 con 0x7fab901022b0 2023-07-12T15:23:27.498 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.923+0000 7fab87fff700 1 -- 172.21.15.69:0/3806876688 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab80011900 con 0x7fab901022b0 2023-07-12T15:23:27.498 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.924+0000 7fab9780a700 1 -- 172.21.15.69:0/3806876688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 msgr2=0x7fab901026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:27.498 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.924+0000 7fab9780a700 1 --2- 172.21.15.69:0/3806876688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 0x7fab901026c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fab800098a0 tx=0x7fab80009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:27.498 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.924+0000 7fab9780a700 1 -- 172.21.15.69:0/3806876688 shutdown_connections 2023-07-12T15:23:27.499 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.924+0000 7fab9780a700 1 --2- 172.21.15.69:0/3806876688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 0x7fab901026c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:27.499 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.924+0000 7fab9780a700 1 -- 172.21.15.69:0/3806876688 >> 172.21.15.69:0/3806876688 conn(0x7fab900fdc30 msgr2=0x7fab90100040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:27.499 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.924+0000 7fab9780a700 1 -- 172.21.15.69:0/3806876688 shutdown_connections 2023-07-12T15:23:27.499 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.924+0000 7fab9780a700 1 -- 172.21.15.69:0/3806876688 wait complete. 2023-07-12T15:23:27.499 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.924+0000 7fab9780a700 1 Processor -- start 2023-07-12T15:23:27.499 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.924+0000 7fab9780a700 1 -- start start 2023-07-12T15:23:27.499 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.925+0000 7fab9780a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 0x7fab90197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:27.500 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.925+0000 7fab9780a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab901981e0 con 0x7fab901022b0 2023-07-12T15:23:27.500 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.925+0000 7fab955a6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 0x7fab90197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:27.500 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.925+0000 7fab955a6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 0x7fab90197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39684/0 (socket says 172.21.15.69:39684) 2023-07-12T15:23:27.500 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.925+0000 7fab955a6700 1 -- 172.21.15.69:0/3896665940 learned_addr learned my addr 172.21.15.69:0/3896665940 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:27.500 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.925+0000 7fab955a6700 1 -- 172.21.15.69:0/3896665940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab80009550 con 0x7fab901022b0 2023-07-12T15:23:27.500 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.925+0000 7fab955a6700 1 --2- 172.21.15.69:0/3896665940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 0x7fab90197ca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fab80006af0 tx=0x7fab80011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:27.501 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.925+0000 7fab867fc700 1 -- 172.21.15.69:0/3896665940 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab800118b0 con 0x7fab901022b0 2023-07-12T15:23:27.501 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.925+0000 7fab867fc700 1 -- 172.21.15.69:0/3896665940 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7fab80003b10 con 0x7fab901022b0 2023-07-12T15:23:27.501 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.925+0000 7fab9780a700 1 -- 172.21.15.69:0/3896665940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab901983e0 con 0x7fab901022b0 2023-07-12T15:23:27.502 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.925+0000 7fab867fc700 1 -- 172.21.15.69:0/3896665940 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab80024c60 con 0x7fab901022b0 2023-07-12T15:23:27.502 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.925+0000 7fab9780a700 1 -- 172.21.15.69:0/3896665940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab90198880 con 0x7fab901022b0 2023-07-12T15:23:27.502 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.926+0000 7fab9780a700 1 -- 172.21.15.69:0/3896665940 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab90062130 con 0x7fab901022b0 2023-07-12T15:23:27.502 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.926+0000 7fab867fc700 1 -- 172.21.15.69:0/3896665940 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) v1 ==== 45030+0+0 (secure 0 0 0) 0x7fab800036a0 con 0x7fab901022b0 2023-07-12T15:23:27.503 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.927+0000 7fab867fc700 1 --2- 172.21.15.69:0/3896665940 >> v2:172.21.15.69:6800/339337620 conn(0x7fab7c0383e0 0x7fab7c03a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:27.503 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.927+0000 7fab867fc700 1 -- 172.21.15.69:0/3896665940 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fab8004bd90 con 0x7fab901022b0 2023-07-12T15:23:27.503 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.927+0000 7fab94da5700 1 --2- 172.21.15.69:0/3896665940 >> v2:172.21.15.69:6800/339337620 conn(0x7fab7c0383e0 0x7fab7c03a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:27.504 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.927+0000 7fab94da5700 1 --2- 172.21.15.69:0/3896665940 >> v2:172.21.15.69:6800/339337620 conn(0x7fab7c0383e0 0x7fab7c03a890 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fab8c006fd0 tx=0x7fab8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:27.504 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:26.931+0000 7fab867fc700 1 -- 172.21.15.69:0/3896665940 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fab80024440 con 0x7fab901022b0 2023-07-12T15:23:27.504 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.119+0000 7fab9780a700 1 -- 172.21.15.69:0/3896665940 --> v2:172.21.15.69:6800/339337620 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fab900ff0d0 con 0x7fab7c0383e0 2023-07-12T15:23:27.505 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.119+0000 7fab867fc700 1 -- 172.21.15.69:0/3896665940 <== mgr.14120 v2:172.21.15.69:6800/339337620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fab900ff0d0 con 0x7fab7c0383e0 2023-07-12T15:23:27.505 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.121+0000 7fab9780a700 1 -- 172.21.15.69:0/3896665940 >> v2:172.21.15.69:6800/339337620 conn(0x7fab7c0383e0 msgr2=0x7fab7c03a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:27.505 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.121+0000 7fab9780a700 1 --2- 172.21.15.69:0/3896665940 >> v2:172.21.15.69:6800/339337620 conn(0x7fab7c0383e0 0x7fab7c03a890 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fab8c006fd0 tx=0x7fab8c006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:27.505 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.121+0000 7fab9780a700 1 -- 172.21.15.69:0/3896665940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 msgr2=0x7fab90197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:27.506 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.121+0000 7fab9780a700 1 --2- 172.21.15.69:0/3896665940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 0x7fab90197ca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fab80006af0 tx=0x7fab80011ea0 comp rx=0 tx=0).stop 2023-07-12T15:23:27.506 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.121+0000 7fab9780a700 1 -- 172.21.15.69:0/3896665940 shutdown_connections 2023-07-12T15:23:27.506 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.121+0000 7fab9780a700 1 --2- 172.21.15.69:0/3896665940 >> v2:172.21.15.69:6800/339337620 conn(0x7fab7c0383e0 0x7fab7c03a890 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:27.507 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.121+0000 7fab9780a700 1 --2- 172.21.15.69:0/3896665940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab901022b0 0x7fab90197ca0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:27.507 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.121+0000 7fab9780a700 1 -- 172.21.15.69:0/3896665940 >> 172.21.15.69:0/3896665940 conn(0x7fab900fdc30 msgr2=0x7fab900fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:27.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.121+0000 7fab9780a700 1 -- 172.21.15.69:0/3896665940 shutdown_connections 2023-07-12T15:23:27.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.121+0000 7fab9780a700 1 -- 172.21.15.69:0/3896665940 wait complete. 2023-07-12T15:23:27.509 INFO:teuthology.orchestra.run.smithi069.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-12T15:23:27.509 INFO:teuthology.orchestra.run.smithi069.stdout:Adding key to root@localhost authorized_keys... 2023-07-12T15:23:27.510 INFO:teuthology.orchestra.run.smithi069.stdout:Adding host smithi069... 2023-07-12T15:23:27.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:27 smithi069 ceph-mon[94241]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:23:28.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:28 smithi069 ceph-mon[94241]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi069", "addr": "172.21.15.69", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:23:28.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:28 smithi069 ceph-mon[94241]: Deploying cephadm binary to smithi069 2023-07-12T15:23:30.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout Added host 'smithi069' with addr '172.21.15.69' 2023-07-12T15:23:30.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.956+0000 7f78782c2700 1 Processor -- start 2023-07-12T15:23:30.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.956+0000 7f78782c2700 1 -- start start 2023-07-12T15:23:30.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.956+0000 7f78782c2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 0x7f7870106e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:30.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.956+0000 7f78782c2700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7870077e80 con 0x7f7870104a50 2023-07-12T15:23:30.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.956+0000 7f787605e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 0x7f7870106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:30.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.956+0000 7f787605e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 0x7f7870106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39688/0 (socket says 172.21.15.69:39688) 2023-07-12T15:23:30.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.956+0000 7f787605e700 1 -- 172.21.15.69:0/2234925829 learned_addr learned my addr 172.21.15.69:0/2234925829 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:30.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.956+0000 7f787605e700 1 -- 172.21.15.69:0/2234925829 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7870107370 con 0x7f7870104a50 2023-07-12T15:23:30.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.957+0000 7f787605e700 1 --2- 172.21.15.69:0/2234925829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 0x7f7870106e30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f786c0098a0 tx=0x7f786c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a4040ac77c88fe server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:30.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.957+0000 7f787505c700 1 -- 172.21.15.69:0/2234925829 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f786c004160 con 0x7f7870104a50 2023-07-12T15:23:30.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.957+0000 7f787505c700 1 -- 172.21.15.69:0/2234925829 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f786c0042c0 con 0x7f7870104a50 2023-07-12T15:23:30.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.957+0000 7f787505c700 1 -- 172.21.15.69:0/2234925829 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f786c011900 con 0x7f7870104a50 2023-07-12T15:23:30.745 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.957+0000 7f78782c2700 1 -- 172.21.15.69:0/2234925829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 msgr2=0x7f7870106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:30.745 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.957+0000 7f78782c2700 1 --2- 172.21.15.69:0/2234925829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 0x7f7870106e30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f786c0098a0 tx=0x7f786c009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:30.745 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.957+0000 7f78782c2700 1 -- 172.21.15.69:0/2234925829 shutdown_connections 2023-07-12T15:23:30.745 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.957+0000 7f78782c2700 1 --2- 172.21.15.69:0/2234925829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 0x7f7870106e30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:30.745 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.957+0000 7f78782c2700 1 -- 172.21.15.69:0/2234925829 >> 172.21.15.69:0/2234925829 conn(0x7f78700fe410 msgr2=0x7f7870100880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:30.745 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.957+0000 7f78782c2700 1 -- 172.21.15.69:0/2234925829 shutdown_connections 2023-07-12T15:23:30.745 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.957+0000 7f78782c2700 1 -- 172.21.15.69:0/2234925829 wait complete. 2023-07-12T15:23:30.746 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.958+0000 7f78782c2700 1 Processor -- start 2023-07-12T15:23:30.746 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.958+0000 7f78782c2700 1 -- start start 2023-07-12T15:23:30.746 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.958+0000 7f78782c2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 0x7f7870197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:30.746 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.958+0000 7f78782c2700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78701981b0 con 0x7f7870104a50 2023-07-12T15:23:30.746 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.958+0000 7f787605e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 0x7f7870197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:30.746 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.958+0000 7f787605e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 0x7f7870197c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39690/0 (socket says 172.21.15.69:39690) 2023-07-12T15:23:30.746 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.958+0000 7f787605e700 1 -- 172.21.15.69:0/1175733372 learned_addr learned my addr 172.21.15.69:0/1175733372 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:30.747 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.958+0000 7f787605e700 1 -- 172.21.15.69:0/1175733372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c009550 con 0x7f7870104a50 2023-07-12T15:23:30.747 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.959+0000 7f787605e700 1 --2- 172.21.15.69:0/1175733372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 0x7f7870197c70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f786c006af0 tx=0x7f786c011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:30.747 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.959+0000 7f78677fe700 1 -- 172.21.15.69:0/1175733372 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f786c011d80 con 0x7f7870104a50 2023-07-12T15:23:30.747 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.959+0000 7f78677fe700 1 -- 172.21.15.69:0/1175733372 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f786c003b10 con 0x7f7870104a50 2023-07-12T15:23:30.747 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.959+0000 7f78782c2700 1 -- 172.21.15.69:0/1175733372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78701983b0 con 0x7f7870104a50 2023-07-12T15:23:30.747 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.959+0000 7f78677fe700 1 -- 172.21.15.69:0/1175733372 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f786c01add0 con 0x7f7870104a50 2023-07-12T15:23:30.747 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.959+0000 7f78782c2700 1 -- 172.21.15.69:0/1175733372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7870198850 con 0x7f7870104a50 2023-07-12T15:23:30.747 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.960+0000 7f78677fe700 1 -- 172.21.15.69:0/1175733372 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) v1 ==== 45030+0+0 (secure 0 0 0) 0x7f786c0036a0 con 0x7f7870104a50 2023-07-12T15:23:30.748 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.960+0000 7f78782c2700 1 -- 172.21.15.69:0/1175733372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78701080b0 con 0x7f7870104a50 2023-07-12T15:23:30.748 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.960+0000 7f78677fe700 1 --2- 172.21.15.69:0/1175733372 >> v2:172.21.15.69:6800/339337620 conn(0x7f785c0383d0 0x7f785c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:30.748 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.960+0000 7f78677fe700 1 -- 172.21.15.69:0/1175733372 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f786c04ba80 con 0x7f7870104a50 2023-07-12T15:23:30.748 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.960+0000 7f787585d700 1 --2- 172.21.15.69:0/1175733372 >> v2:172.21.15.69:6800/339337620 conn(0x7f785c0383d0 0x7f785c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:30.748 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.960+0000 7f787585d700 1 --2- 172.21.15.69:0/1175733372 >> v2:172.21.15.69:6800/339337620 conn(0x7f785c0383d0 0x7f785c03a880 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7860006fd0 tx=0x7f7860006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:30.749 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:27.964+0000 7f78677fe700 1 -- 172.21.15.69:0/1175733372 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f786c003950 con 0x7f7870104a50 2023-07-12T15:23:30.749 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:28.153+0000 7f78782c2700 1 -- 172.21.15.69:0/1175733372 --> v2:172.21.15.69:6800/339337620 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi069", "addr": "172.21.15.69", "target": ["mon-mgr", ""]}) v1 -- 0x7f78700ff960 con 0x7f785c0383d0 2023-07-12T15:23:30.749 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:30.362+0000 7f78677fe700 1 -- 172.21.15.69:0/1175733372 <== mgr.14120 v2:172.21.15.69:6800/339337620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f78700ff960 con 0x7f785c0383d0 2023-07-12T15:23:30.749 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:30.365+0000 7f78782c2700 1 -- 172.21.15.69:0/1175733372 >> v2:172.21.15.69:6800/339337620 conn(0x7f785c0383d0 msgr2=0x7f785c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:30.749 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:30.365+0000 7f78782c2700 1 --2- 172.21.15.69:0/1175733372 >> v2:172.21.15.69:6800/339337620 conn(0x7f785c0383d0 0x7f785c03a880 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7860006fd0 tx=0x7f7860006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:30.749 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:30.366+0000 7f78782c2700 1 -- 172.21.15.69:0/1175733372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 msgr2=0x7f7870197c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:30.749 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:30.366+0000 7f78782c2700 1 --2- 172.21.15.69:0/1175733372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 0x7f7870197c70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f786c006af0 tx=0x7f786c011980 comp rx=0 tx=0).stop 2023-07-12T15:23:30.750 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:30.366+0000 7f78782c2700 1 -- 172.21.15.69:0/1175733372 shutdown_connections 2023-07-12T15:23:30.750 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:30.366+0000 7f78782c2700 1 --2- 172.21.15.69:0/1175733372 >> v2:172.21.15.69:6800/339337620 conn(0x7f785c0383d0 0x7f785c03a880 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:30.750 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:30.366+0000 7f78782c2700 1 --2- 172.21.15.69:0/1175733372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7870104a50 0x7f7870197c70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:30.750 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:30.366+0000 7f78782c2700 1 -- 172.21.15.69:0/1175733372 >> 172.21.15.69:0/1175733372 conn(0x7f78700fe410 msgr2=0x7f78700ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:30.750 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:30.366+0000 7f78782c2700 1 -- 172.21.15.69:0/1175733372 shutdown_connections 2023-07-12T15:23:30.750 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:30.366+0000 7f78782c2700 1 -- 172.21.15.69:0/1175733372 wait complete. 2023-07-12T15:23:30.750 INFO:teuthology.orchestra.run.smithi069.stdout:Deploying unmanaged mon service... 2023-07-12T15:23:31.641 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:31 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:31.642 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:31 smithi069 ceph-mon[94241]: Added host smithi069 2023-07-12T15:23:31.642 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:31 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:23:32.089 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-12T15:23:32.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.220+0000 7f433d5d2700 1 Processor -- start 2023-07-12T15:23:32.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.220+0000 7f433d5d2700 1 -- start start 2023-07-12T15:23:32.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.220+0000 7f433d5d2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 0x7f4338107050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:32.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.220+0000 7f433d5d2700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4338077e80 con 0x7f4338104c70 2023-07-12T15:23:32.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.220+0000 7f4336ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 0x7f4338107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:32.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.220+0000 7f4336ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 0x7f4338107050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39714/0 (socket says 172.21.15.69:39714) 2023-07-12T15:23:32.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.220+0000 7f4336ffd700 1 -- 172.21.15.69:0/490236437 learned_addr learned my addr 172.21.15.69:0/490236437 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:32.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.220+0000 7f4336ffd700 1 -- 172.21.15.69:0/490236437 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4338107590 con 0x7f4338104c70 2023-07-12T15:23:32.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.220+0000 7f4336ffd700 1 --2- 172.21.15.69:0/490236437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 0x7f4338107050 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f43280098a0 tx=0x7f4328009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902b5314d80c227f server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:32.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.221+0000 7f4335ffb700 1 -- 172.21.15.69:0/490236437 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4328004160 con 0x7f4338104c70 2023-07-12T15:23:32.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.221+0000 7f4335ffb700 1 -- 172.21.15.69:0/490236437 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f43280042c0 con 0x7f4338104c70 2023-07-12T15:23:32.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.221+0000 7f4335ffb700 1 -- 172.21.15.69:0/490236437 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4328011900 con 0x7f4338104c70 2023-07-12T15:23:32.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.221+0000 7f433d5d2700 1 -- 172.21.15.69:0/490236437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 msgr2=0x7f4338107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:32.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.221+0000 7f433d5d2700 1 --2- 172.21.15.69:0/490236437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 0x7f4338107050 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f43280098a0 tx=0x7f4328009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:32.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.221+0000 7f433d5d2700 1 -- 172.21.15.69:0/490236437 shutdown_connections 2023-07-12T15:23:32.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.221+0000 7f433d5d2700 1 --2- 172.21.15.69:0/490236437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 0x7f4338107050 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:32.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.221+0000 7f433d5d2700 1 -- 172.21.15.69:0/490236437 >> 172.21.15.69:0/490236437 conn(0x7f43380fe630 msgr2=0x7f4338100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:32.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.221+0000 7f433d5d2700 1 -- 172.21.15.69:0/490236437 shutdown_connections 2023-07-12T15:23:32.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.221+0000 7f433d5d2700 1 -- 172.21.15.69:0/490236437 wait complete. 2023-07-12T15:23:32.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.222+0000 7f433d5d2700 1 Processor -- start 2023-07-12T15:23:32.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.222+0000 7f433d5d2700 1 -- start start 2023-07-12T15:23:32.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.222+0000 7f433d5d2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 0x7f4338197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:32.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.222+0000 7f433d5d2700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4338198300 con 0x7f4338104c70 2023-07-12T15:23:32.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.222+0000 7f4336ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 0x7f4338197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:32.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.222+0000 7f4336ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 0x7f4338197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39716/0 (socket says 172.21.15.69:39716) 2023-07-12T15:23:32.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.222+0000 7f4336ffd700 1 -- 172.21.15.69:0/328787263 learned_addr learned my addr 172.21.15.69:0/328787263 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:32.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.223+0000 7f4336ffd700 1 -- 172.21.15.69:0/328787263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4328009550 con 0x7f4338104c70 2023-07-12T15:23:32.094 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.223+0000 7f4336ffd700 1 --2- 172.21.15.69:0/328787263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 0x7f4338197dc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4328006af0 tx=0x7f4328011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:32.094 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.223+0000 7f431ffff700 1 -- 172.21.15.69:0/328787263 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4328011d80 con 0x7f4338104c70 2023-07-12T15:23:32.094 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.223+0000 7f431ffff700 1 -- 172.21.15.69:0/328787263 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1027+0+0 (secure 0 0 0) 0x7f4328003b10 con 0x7f4338104c70 2023-07-12T15:23:32.094 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.223+0000 7f433d5d2700 1 -- 172.21.15.69:0/328787263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4338198500 con 0x7f4338104c70 2023-07-12T15:23:32.094 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.223+0000 7f431ffff700 1 -- 172.21.15.69:0/328787263 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f432801adb0 con 0x7f4338104c70 2023-07-12T15:23:32.094 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.223+0000 7f433d5d2700 1 -- 172.21.15.69:0/328787263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43381989a0 con 0x7f4338104c70 2023-07-12T15:23:32.094 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.224+0000 7f431ffff700 1 -- 172.21.15.69:0/328787263 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) v1 ==== 45030+0+0 (secure 0 0 0) 0x7f43280243f0 con 0x7f4338104c70 2023-07-12T15:23:32.094 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.224+0000 7f431ffff700 1 --2- 172.21.15.69:0/328787263 >> v2:172.21.15.69:6800/339337620 conn(0x7f43200383d0 0x7f432003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:32.095 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.224+0000 7f431ffff700 1 -- 172.21.15.69:0/328787263 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f43280036c0 con 0x7f4338104c70 2023-07-12T15:23:32.095 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.224+0000 7f433d5d2700 1 -- 172.21.15.69:0/328787263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4338062130 con 0x7f4338104c70 2023-07-12T15:23:32.095 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.224+0000 7f43367fc700 1 --2- 172.21.15.69:0/328787263 >> v2:172.21.15.69:6800/339337620 conn(0x7f43200383d0 0x7f432003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:32.095 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.225+0000 7f43367fc700 1 --2- 172.21.15.69:0/328787263 >> v2:172.21.15.69:6800/339337620 conn(0x7f43200383d0 0x7f432003a880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f432c006fd0 tx=0x7f432c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:32.095 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.231+0000 7f431ffff700 1 -- 172.21.15.69:0/328787263 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f43280246a0 con 0x7f4338104c70 2023-07-12T15:23:32.095 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.441+0000 7f433d5d2700 1 -- 172.21.15.69:0/328787263 --> v2:172.21.15.69:6800/339337620 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f433819b360 con 0x7f43200383d0 2023-07-12T15:23:32.095 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.446+0000 7f431ffff700 1 -- 172.21.15.69:0/328787263 <== mgr.14120 v2:172.21.15.69:6800/339337620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f433819b360 con 0x7f43200383d0 2023-07-12T15:23:32.096 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.448+0000 7f433d5d2700 1 -- 172.21.15.69:0/328787263 >> v2:172.21.15.69:6800/339337620 conn(0x7f43200383d0 msgr2=0x7f432003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:32.096 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.448+0000 7f433d5d2700 1 --2- 172.21.15.69:0/328787263 >> v2:172.21.15.69:6800/339337620 conn(0x7f43200383d0 0x7f432003a880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f432c006fd0 tx=0x7f432c006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:32.096 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.448+0000 7f433d5d2700 1 -- 172.21.15.69:0/328787263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 msgr2=0x7f4338197dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:32.096 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.448+0000 7f433d5d2700 1 --2- 172.21.15.69:0/328787263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 0x7f4338197dc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4328006af0 tx=0x7f4328011980 comp rx=0 tx=0).stop 2023-07-12T15:23:32.096 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.448+0000 7f433d5d2700 1 -- 172.21.15.69:0/328787263 shutdown_connections 2023-07-12T15:23:32.096 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.448+0000 7f433d5d2700 1 --2- 172.21.15.69:0/328787263 >> v2:172.21.15.69:6800/339337620 conn(0x7f43200383d0 0x7f432003a880 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:32.096 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.448+0000 7f433d5d2700 1 --2- 172.21.15.69:0/328787263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4338104c70 0x7f4338197dc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:32.097 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.448+0000 7f433d5d2700 1 -- 172.21.15.69:0/328787263 >> 172.21.15.69:0/328787263 conn(0x7f43380fe630 msgr2=0x7f43380ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:32.097 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.448+0000 7f433d5d2700 1 -- 172.21.15.69:0/328787263 shutdown_connections 2023-07-12T15:23:32.097 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:31.448+0000 7f433d5d2700 1 -- 172.21.15.69:0/328787263 wait complete. 2023-07-12T15:23:32.097 INFO:teuthology.orchestra.run.smithi069.stdout:Deploying unmanaged mgr service... 2023-07-12T15:23:32.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:32 smithi069 ceph-mon[94241]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:23:32.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:32 smithi069 ceph-mon[94241]: Saving service mon spec with placement count:5 2023-07-12T15:23:32.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:32 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:32.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:32 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:33.134 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-12T15:23:33.135 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.556+0000 7fd24f3a2700 1 Processor -- start 2023-07-12T15:23:33.135 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.557+0000 7fd24f3a2700 1 -- start start 2023-07-12T15:23:33.135 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.557+0000 7fd24f3a2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 0x7fd2481037c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:33.135 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.557+0000 7fd24f3a2700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd248103d00 con 0x7fd2481033b0 2023-07-12T15:23:33.135 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.557+0000 7fd24d13e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 0x7fd2481037c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:33.135 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.557+0000 7fd24d13e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 0x7fd2481037c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39720/0 (socket says 172.21.15.69:39720) 2023-07-12T15:23:33.136 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.557+0000 7fd24d13e700 1 -- 172.21.15.69:0/2748783891 learned_addr learned my addr 172.21.15.69:0/2748783891 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:33.136 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.557+0000 7fd24d13e700 1 -- 172.21.15.69:0/2748783891 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd248103e40 con 0x7fd2481033b0 2023-07-12T15:23:33.136 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.558+0000 7fd24d13e700 1 --2- 172.21.15.69:0/2748783891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 0x7fd2481037c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2440098a0 tx=0x7fd244009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d97a54167681281 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:33.136 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.558+0000 7fd23ffff700 1 -- 172.21.15.69:0/2748783891 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd244004160 con 0x7fd2481033b0 2023-07-12T15:23:33.136 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.558+0000 7fd23ffff700 1 -- 172.21.15.69:0/2748783891 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fd2440042c0 con 0x7fd2481033b0 2023-07-12T15:23:33.136 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.558+0000 7fd23ffff700 1 -- 172.21.15.69:0/2748783891 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd2440118c0 con 0x7fd2481033b0 2023-07-12T15:23:33.136 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.558+0000 7fd24f3a2700 1 -- 172.21.15.69:0/2748783891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 msgr2=0x7fd2481037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:33.136 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.558+0000 7fd24f3a2700 1 --2- 172.21.15.69:0/2748783891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 0x7fd2481037c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2440098a0 tx=0x7fd244009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:33.137 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.558+0000 7fd24f3a2700 1 -- 172.21.15.69:0/2748783891 shutdown_connections 2023-07-12T15:23:33.137 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.558+0000 7fd24f3a2700 1 --2- 172.21.15.69:0/2748783891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 0x7fd2481037c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:33.137 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.558+0000 7fd24f3a2700 1 -- 172.21.15.69:0/2748783891 >> 172.21.15.69:0/2748783891 conn(0x7fd2480fdc60 msgr2=0x7fd2481000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:33.137 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.559+0000 7fd24f3a2700 1 -- 172.21.15.69:0/2748783891 shutdown_connections 2023-07-12T15:23:33.137 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.559+0000 7fd24f3a2700 1 -- 172.21.15.69:0/2748783891 wait complete. 2023-07-12T15:23:33.137 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.559+0000 7fd24f3a2700 1 Processor -- start 2023-07-12T15:23:33.137 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.559+0000 7fd24f3a2700 1 -- start start 2023-07-12T15:23:33.138 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.559+0000 7fd24f3a2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 0x7fd248102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:33.138 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.559+0000 7fd24f3a2700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24819a920 con 0x7fd2481033b0 2023-07-12T15:23:33.138 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.559+0000 7fd24d13e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 0x7fd248102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:33.138 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.559+0000 7fd24d13e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 0x7fd248102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39722/0 (socket says 172.21.15.69:39722) 2023-07-12T15:23:33.138 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.559+0000 7fd24d13e700 1 -- 172.21.15.69:0/675028934 learned_addr learned my addr 172.21.15.69:0/675028934 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:33.138 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.560+0000 7fd24d13e700 1 -- 172.21.15.69:0/675028934 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd244009550 con 0x7fd2481033b0 2023-07-12T15:23:33.138 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.560+0000 7fd24d13e700 1 --2- 172.21.15.69:0/675028934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 0x7fd248102b30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd2440046c0 tx=0x7fd2440119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:33.139 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.560+0000 7fd23e7fc700 1 -- 172.21.15.69:0/675028934 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd2440036a0 con 0x7fd2481033b0 2023-07-12T15:23:33.139 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.560+0000 7fd23e7fc700 1 -- 172.21.15.69:0/675028934 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fd244003800 con 0x7fd2481033b0 2023-07-12T15:23:33.139 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.560+0000 7fd23e7fc700 1 -- 172.21.15.69:0/675028934 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd24401a430 con 0x7fd2481033b0 2023-07-12T15:23:33.139 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.560+0000 7fd24f3a2700 1 -- 172.21.15.69:0/675028934 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24819aa60 con 0x7fd2481033b0 2023-07-12T15:23:33.139 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.560+0000 7fd24f3a2700 1 -- 172.21.15.69:0/675028934 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd24819af00 con 0x7fd2481033b0 2023-07-12T15:23:33.139 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.561+0000 7fd23e7fc700 1 -- 172.21.15.69:0/675028934 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) v1 ==== 45030+0+0 (secure 0 0 0) 0x7fd24401a590 con 0x7fd2481033b0 2023-07-12T15:23:33.139 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.561+0000 7fd23e7fc700 1 --2- 172.21.15.69:0/675028934 >> v2:172.21.15.69:6800/339337620 conn(0x7fd2340383d0 0x7fd23403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:33.139 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.561+0000 7fd24f3a2700 1 -- 172.21.15.69:0/675028934 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd248062130 con 0x7fd2481033b0 2023-07-12T15:23:33.140 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.561+0000 7fd23e7fc700 1 -- 172.21.15.69:0/675028934 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fd24404be00 con 0x7fd2481033b0 2023-07-12T15:23:33.140 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.561+0000 7fd24c93d700 1 --2- 172.21.15.69:0/675028934 >> v2:172.21.15.69:6800/339337620 conn(0x7fd2340383d0 0x7fd23403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:33.140 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.562+0000 7fd24c93d700 1 --2- 172.21.15.69:0/675028934 >> v2:172.21.15.69:6800/339337620 conn(0x7fd2340383d0 0x7fd23403a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd238006fd0 tx=0x7fd238006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:33.140 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.566+0000 7fd23e7fc700 1 -- 172.21.15.69:0/675028934 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd244024430 con 0x7fd2481033b0 2023-07-12T15:23:33.140 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.775+0000 7fd24f3a2700 1 -- 172.21.15.69:0/675028934 --> v2:172.21.15.69:6800/339337620 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fd2480ff0a0 con 0x7fd2340383d0 2023-07-12T15:23:33.140 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.779+0000 7fd23e7fc700 1 -- 172.21.15.69:0/675028934 <== mgr.14120 v2:172.21.15.69:6800/339337620 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd2480ff0a0 con 0x7fd2340383d0 2023-07-12T15:23:33.140 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.782+0000 7fd24f3a2700 1 -- 172.21.15.69:0/675028934 >> v2:172.21.15.69:6800/339337620 conn(0x7fd2340383d0 msgr2=0x7fd23403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:33.141 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.782+0000 7fd24f3a2700 1 --2- 172.21.15.69:0/675028934 >> v2:172.21.15.69:6800/339337620 conn(0x7fd2340383d0 0x7fd23403a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd238006fd0 tx=0x7fd238006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:33.141 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.782+0000 7fd24f3a2700 1 -- 172.21.15.69:0/675028934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 msgr2=0x7fd248102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:33.141 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.782+0000 7fd24f3a2700 1 --2- 172.21.15.69:0/675028934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 0x7fd248102b30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd2440046c0 tx=0x7fd2440119d0 comp rx=0 tx=0).stop 2023-07-12T15:23:33.141 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.782+0000 7fd24f3a2700 1 -- 172.21.15.69:0/675028934 shutdown_connections 2023-07-12T15:23:33.141 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.782+0000 7fd24f3a2700 1 --2- 172.21.15.69:0/675028934 >> v2:172.21.15.69:6800/339337620 conn(0x7fd2340383d0 0x7fd23403a880 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:33.141 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.782+0000 7fd24f3a2700 1 --2- 172.21.15.69:0/675028934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2481033b0 0x7fd248102b30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:33.141 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.782+0000 7fd24f3a2700 1 -- 172.21.15.69:0/675028934 >> 172.21.15.69:0/675028934 conn(0x7fd2480fdc60 msgr2=0x7fd2480fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:33.142 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.782+0000 7fd24f3a2700 1 -- 172.21.15.69:0/675028934 shutdown_connections 2023-07-12T15:23:33.142 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:32.782+0000 7fd24f3a2700 1 -- 172.21.15.69:0/675028934 wait complete. 2023-07-12T15:23:33.849 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:33 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:33.850 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:33 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:34.177 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.613+0000 7fb175f46700 1 Processor -- start 2023-07-12T15:23:34.177 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.613+0000 7fb175f46700 1 -- start start 2023-07-12T15:23:34.178 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.614+0000 7fb175f46700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 0x7fb1701032e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:34.178 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.614+0000 7fb175f46700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb170103820 con 0x7fb170102ed0 2023-07-12T15:23:34.178 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.614+0000 7fb16f7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 0x7fb1701032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:34.178 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.614+0000 7fb16f7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 0x7fb1701032e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39726/0 (socket says 172.21.15.69:39726) 2023-07-12T15:23:34.178 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.614+0000 7fb16f7fe700 1 -- 172.21.15.69:0/2824559425 learned_addr learned my addr 172.21.15.69:0/2824559425 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:34.178 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.614+0000 7fb16f7fe700 1 -- 172.21.15.69:0/2824559425 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb170103960 con 0x7fb170102ed0 2023-07-12T15:23:34.178 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.614+0000 7fb16f7fe700 1 --2- 172.21.15.69:0/2824559425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 0x7fb1701032e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb1600098a0 tx=0x7fb160009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1302dfebf7a4030 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:34.179 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.614+0000 7fb16effd700 1 -- 172.21.15.69:0/2824559425 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb160004160 con 0x7fb170102ed0 2023-07-12T15:23:34.179 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.614+0000 7fb16effd700 1 -- 172.21.15.69:0/2824559425 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fb1600042c0 con 0x7fb170102ed0 2023-07-12T15:23:34.179 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.614+0000 7fb16effd700 1 -- 172.21.15.69:0/2824559425 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1600118c0 con 0x7fb170102ed0 2023-07-12T15:23:34.179 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.615+0000 7fb175f46700 1 -- 172.21.15.69:0/2824559425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 msgr2=0x7fb1701032e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:34.179 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.615+0000 7fb175f46700 1 --2- 172.21.15.69:0/2824559425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 0x7fb1701032e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb1600098a0 tx=0x7fb160009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:34.179 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.615+0000 7fb175f46700 1 -- 172.21.15.69:0/2824559425 shutdown_connections 2023-07-12T15:23:34.179 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.615+0000 7fb175f46700 1 --2- 172.21.15.69:0/2824559425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 0x7fb1701032e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:34.180 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.615+0000 7fb175f46700 1 -- 172.21.15.69:0/2824559425 >> 172.21.15.69:0/2824559425 conn(0x7fb1700fe540 msgr2=0x7fb170100950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:34.180 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.615+0000 7fb175f46700 1 -- 172.21.15.69:0/2824559425 shutdown_connections 2023-07-12T15:23:34.180 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.615+0000 7fb175f46700 1 -- 172.21.15.69:0/2824559425 wait complete. 2023-07-12T15:23:34.180 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.616+0000 7fb175f46700 1 Processor -- start 2023-07-12T15:23:34.180 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.616+0000 7fb175f46700 1 -- start start 2023-07-12T15:23:34.180 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.616+0000 7fb175f46700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 0x7fb170199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:34.180 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.616+0000 7fb175f46700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17019a350 con 0x7fb170102ed0 2023-07-12T15:23:34.181 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.616+0000 7fb16f7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 0x7fb170199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:34.181 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.616+0000 7fb16f7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 0x7fb170199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39728/0 (socket says 172.21.15.69:39728) 2023-07-12T15:23:34.181 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.616+0000 7fb16f7fe700 1 -- 172.21.15.69:0/2802495930 learned_addr learned my addr 172.21.15.69:0/2802495930 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:34.181 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.616+0000 7fb16f7fe700 1 -- 172.21.15.69:0/2802495930 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb160009550 con 0x7fb170102ed0 2023-07-12T15:23:34.181 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.617+0000 7fb16f7fe700 1 --2- 172.21.15.69:0/2802495930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 0x7fb170199e10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb1600046c0 tx=0x7fb160011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:34.181 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.617+0000 7fb16d7fa700 1 -- 172.21.15.69:0/2802495930 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1600036a0 con 0x7fb170102ed0 2023-07-12T15:23:34.181 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.617+0000 7fb16d7fa700 1 -- 172.21.15.69:0/2802495930 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fb160003800 con 0x7fb170102ed0 2023-07-12T15:23:34.181 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.617+0000 7fb175f46700 1 -- 172.21.15.69:0/2802495930 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb17019a550 con 0x7fb170102ed0 2023-07-12T15:23:34.182 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.617+0000 7fb16d7fa700 1 -- 172.21.15.69:0/2802495930 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb16001aea0 con 0x7fb170102ed0 2023-07-12T15:23:34.182 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.617+0000 7fb175f46700 1 -- 172.21.15.69:0/2802495930 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17019a9f0 con 0x7fb170102ed0 2023-07-12T15:23:34.182 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.618+0000 7fb16d7fa700 1 -- 172.21.15.69:0/2802495930 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) v1 ==== 45030+0+0 (secure 0 0 0) 0x7fb16002d440 con 0x7fb170102ed0 2023-07-12T15:23:34.182 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.618+0000 7fb16d7fa700 1 --2- 172.21.15.69:0/2802495930 >> v2:172.21.15.69:6800/339337620 conn(0x7fb15c0383d0 0x7fb15c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:34.182 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.618+0000 7fb16d7fa700 1 -- 172.21.15.69:0/2802495930 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fb16004bd70 con 0x7fb170102ed0 2023-07-12T15:23:34.182 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.618+0000 7fb166ffd700 1 --2- 172.21.15.69:0/2802495930 >> v2:172.21.15.69:6800/339337620 conn(0x7fb15c0383d0 0x7fb15c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:34.182 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.618+0000 7fb175f46700 1 -- 172.21.15.69:0/2802495930 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb17018f800 con 0x7fb170102ed0 2023-07-12T15:23:34.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.618+0000 7fb166ffd700 1 --2- 172.21.15.69:0/2802495930 >> v2:172.21.15.69:6800/339337620 conn(0x7fb15c0383d0 0x7fb15c03a880 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb158006fd0 tx=0x7fb158006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:34.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.623+0000 7fb16d7fa700 1 -- 172.21.15.69:0/2802495930 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb160018940 con 0x7fb170102ed0 2023-07-12T15:23:34.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.817+0000 7fb175f46700 1 -- 172.21.15.69:0/2802495930 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fb170104d70 con 0x7fb170102ed0 2023-07-12T15:23:34.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.820+0000 7fb16d7fa700 1 -- 172.21.15.69:0/2802495930 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fb160018350 con 0x7fb170102ed0 2023-07-12T15:23:34.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.822+0000 7fb175f46700 1 -- 172.21.15.69:0/2802495930 >> v2:172.21.15.69:6800/339337620 conn(0x7fb15c0383d0 msgr2=0x7fb15c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:34.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.822+0000 7fb175f46700 1 --2- 172.21.15.69:0/2802495930 >> v2:172.21.15.69:6800/339337620 conn(0x7fb15c0383d0 0x7fb15c03a880 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb158006fd0 tx=0x7fb158006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:34.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.822+0000 7fb175f46700 1 -- 172.21.15.69:0/2802495930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 msgr2=0x7fb170199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:34.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.822+0000 7fb175f46700 1 --2- 172.21.15.69:0/2802495930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 0x7fb170199e10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb1600046c0 tx=0x7fb160011920 comp rx=0 tx=0).stop 2023-07-12T15:23:34.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.822+0000 7fb175f46700 1 -- 172.21.15.69:0/2802495930 shutdown_connections 2023-07-12T15:23:34.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.822+0000 7fb175f46700 1 --2- 172.21.15.69:0/2802495930 >> v2:172.21.15.69:6800/339337620 conn(0x7fb15c0383d0 0x7fb15c03a880 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:34.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.822+0000 7fb175f46700 1 --2- 172.21.15.69:0/2802495930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb170102ed0 0x7fb170199e10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:34.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.822+0000 7fb175f46700 1 -- 172.21.15.69:0/2802495930 >> 172.21.15.69:0/2802495930 conn(0x7fb1700fe540 msgr2=0x7fb17018c940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:34.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.822+0000 7fb175f46700 1 -- 172.21.15.69:0/2802495930 shutdown_connections 2023-07-12T15:23:34.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:33.822+0000 7fb175f46700 1 -- 172.21.15.69:0/2802495930 wait complete. 2023-07-12T15:23:34.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:34 smithi069 ceph-mon[94241]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:23:34.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:34 smithi069 ceph-mon[94241]: Saving service mgr spec with placement count:2 2023-07-12T15:23:34.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2802495930' entity='client.admin' 2023-07-12T15:23:35.475 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.884+0000 7f02aaf18700 1 Processor -- start 2023-07-12T15:23:35.476 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.884+0000 7f02aaf18700 1 -- start start 2023-07-12T15:23:35.476 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.884+0000 7f02aaf18700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 0x7f02a4103260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:35.476 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.884+0000 7f02aaf18700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a41037a0 con 0x7f02a4102e50 2023-07-12T15:23:35.476 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.884+0000 7f02a8cb4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 0x7f02a4103260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:35.477 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.884+0000 7f02a8cb4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 0x7f02a4103260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39732/0 (socket says 172.21.15.69:39732) 2023-07-12T15:23:35.477 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.884+0000 7f02a8cb4700 1 -- 172.21.15.69:0/2509296626 learned_addr learned my addr 172.21.15.69:0/2509296626 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:35.477 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.884+0000 7f02a8cb4700 1 -- 172.21.15.69:0/2509296626 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a41038e0 con 0x7f02a4102e50 2023-07-12T15:23:35.477 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.885+0000 7f02a8cb4700 1 --2- 172.21.15.69:0/2509296626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 0x7f02a4103260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f02980098a0 tx=0x7f0298009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55dfde6261b716c server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:35.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.885+0000 7f02a37fe700 1 -- 172.21.15.69:0/2509296626 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0298004160 con 0x7f02a4102e50 2023-07-12T15:23:35.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.885+0000 7f02a37fe700 1 -- 172.21.15.69:0/2509296626 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f02980042c0 con 0x7f02a4102e50 2023-07-12T15:23:35.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.885+0000 7f02a37fe700 1 -- 172.21.15.69:0/2509296626 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f02980118c0 con 0x7f02a4102e50 2023-07-12T15:23:35.478 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.885+0000 7f02aaf18700 1 -- 172.21.15.69:0/2509296626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 msgr2=0x7f02a4103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:35.479 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.885+0000 7f02aaf18700 1 --2- 172.21.15.69:0/2509296626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 0x7f02a4103260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f02980098a0 tx=0x7f0298009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:35.479 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.885+0000 7f02aaf18700 1 -- 172.21.15.69:0/2509296626 shutdown_connections 2023-07-12T15:23:35.479 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.885+0000 7f02aaf18700 1 --2- 172.21.15.69:0/2509296626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 0x7f02a4103260 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:35.479 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.885+0000 7f02aaf18700 1 -- 172.21.15.69:0/2509296626 >> 172.21.15.69:0/2509296626 conn(0x7f02a40fe410 msgr2=0x7f02a4100880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:35.479 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.886+0000 7f02aaf18700 1 -- 172.21.15.69:0/2509296626 shutdown_connections 2023-07-12T15:23:35.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.886+0000 7f02aaf18700 1 -- 172.21.15.69:0/2509296626 wait complete. 2023-07-12T15:23:35.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.886+0000 7f02aaf18700 1 Processor -- start 2023-07-12T15:23:35.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.886+0000 7f02aaf18700 1 -- start start 2023-07-12T15:23:35.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.887+0000 7f02aaf18700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 0x7f02a4196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:35.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.887+0000 7f02aaf18700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a4197080 con 0x7f02a4102e50 2023-07-12T15:23:35.480 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.887+0000 7f02a8cb4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 0x7f02a4196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:35.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.887+0000 7f02a8cb4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 0x7f02a4196b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39734/0 (socket says 172.21.15.69:39734) 2023-07-12T15:23:35.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.887+0000 7f02a8cb4700 1 -- 172.21.15.69:0/2229104914 learned_addr learned my addr 172.21.15.69:0/2229104914 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:35.481 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.887+0000 7f02a8cb4700 1 -- 172.21.15.69:0/2229104914 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0298009550 con 0x7f02a4102e50 2023-07-12T15:23:35.482 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.887+0000 7f02a8cb4700 1 --2- 172.21.15.69:0/2229104914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 0x7f02a4196b40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f02980046c0 tx=0x7f0298011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:35.482 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.887+0000 7f02a1ffb700 1 -- 172.21.15.69:0/2229104914 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f02980036a0 con 0x7f02a4102e50 2023-07-12T15:23:35.482 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.887+0000 7f02a1ffb700 1 -- 172.21.15.69:0/2229104914 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f0298003800 con 0x7f02a4102e50 2023-07-12T15:23:35.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.887+0000 7f02a1ffb700 1 -- 172.21.15.69:0/2229104914 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0298024e70 con 0x7f02a4102e50 2023-07-12T15:23:35.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.887+0000 7f02aaf18700 1 -- 172.21.15.69:0/2229104914 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a4197280 con 0x7f02a4102e50 2023-07-12T15:23:35.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.888+0000 7f02aaf18700 1 -- 172.21.15.69:0/2229104914 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a4197720 con 0x7f02a4102e50 2023-07-12T15:23:35.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.888+0000 7f02a1ffb700 1 -- 172.21.15.69:0/2229104914 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) v1 ==== 45030+0+0 (secure 0 0 0) 0x7f0298024430 con 0x7f02a4102e50 2023-07-12T15:23:35.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.888+0000 7f02aaf18700 1 -- 172.21.15.69:0/2229104914 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a4062130 con 0x7f02a4102e50 2023-07-12T15:23:35.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.889+0000 7f02a1ffb700 1 --2- 172.21.15.69:0/2229104914 >> v2:172.21.15.69:6800/339337620 conn(0x7f0294038330 0x7f029403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:35.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.889+0000 7f02a1ffb700 1 -- 172.21.15.69:0/2229104914 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f029804bb20 con 0x7f02a4102e50 2023-07-12T15:23:35.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.889+0000 7f02a3fff700 1 --2- 172.21.15.69:0/2229104914 >> v2:172.21.15.69:6800/339337620 conn(0x7f0294038330 0x7f029403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:35.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.889+0000 7f02a3fff700 1 --2- 172.21.15.69:0/2229104914 >> v2:172.21.15.69:6800/339337620 conn(0x7f0294038330 0x7f029403a7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0290006fd0 tx=0x7f0290006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:35.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:34.897+0000 7f02a1ffb700 1 -- 172.21.15.69:0/2229104914 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f02980189a0 con 0x7f02a4102e50 2023-07-12T15:23:35.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.101+0000 7f02aaf18700 1 -- 172.21.15.69:0/2229104914 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f02a419a6b0 con 0x7f02a4102e50 2023-07-12T15:23:35.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.104+0000 7f02a1ffb700 1 -- 172.21.15.69:0/2229104914 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f0298018350 con 0x7f02a4102e50 2023-07-12T15:23:35.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.106+0000 7f02aaf18700 1 -- 172.21.15.69:0/2229104914 >> v2:172.21.15.69:6800/339337620 conn(0x7f0294038330 msgr2=0x7f029403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:35.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.106+0000 7f02aaf18700 1 --2- 172.21.15.69:0/2229104914 >> v2:172.21.15.69:6800/339337620 conn(0x7f0294038330 0x7f029403a7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0290006fd0 tx=0x7f0290006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:35.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.107+0000 7f02aaf18700 1 -- 172.21.15.69:0/2229104914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 msgr2=0x7f02a4196b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:35.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.107+0000 7f02aaf18700 1 --2- 172.21.15.69:0/2229104914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 0x7f02a4196b40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f02980046c0 tx=0x7f0298011920 comp rx=0 tx=0).stop 2023-07-12T15:23:35.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.107+0000 7f02aaf18700 1 -- 172.21.15.69:0/2229104914 shutdown_connections 2023-07-12T15:23:35.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.107+0000 7f02aaf18700 1 --2- 172.21.15.69:0/2229104914 >> v2:172.21.15.69:6800/339337620 conn(0x7f0294038330 0x7f029403a7e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:35.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.107+0000 7f02aaf18700 1 --2- 172.21.15.69:0/2229104914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a4102e50 0x7f02a4196b40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:35.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.107+0000 7f02aaf18700 1 -- 172.21.15.69:0/2229104914 >> 172.21.15.69:0/2229104914 conn(0x7f02a40fe410 msgr2=0x7f02a40ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:35.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.107+0000 7f02aaf18700 1 -- 172.21.15.69:0/2229104914 shutdown_connections 2023-07-12T15:23:35.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.107+0000 7f02aaf18700 1 -- 172.21.15.69:0/2229104914 wait complete. 2023-07-12T15:23:35.487 INFO:teuthology.orchestra.run.smithi069.stdout:Enabling the dashboard module... 2023-07-12T15:23:36.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:35 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:36.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2229104914' entity='client.admin' 2023-07-12T15:23:36.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:35 smithi069 ceph-mon[94241]: from='mgr.14120 172.21.15.69:0/3625803196' entity='mgr.y' 2023-07-12T15:23:36.842 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/476434071' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-12T15:23:37.160 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ignoring --setuser ceph since I am not root 2023-07-12T15:23:37.160 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ignoring --setgroup ceph since I am not root 2023-07-12T15:23:37.160 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:36.988+0000 7fe0506d4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-12T15:23:37.161 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:37 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:37.067+0000 7fe0506d4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-12T15:23:37.161 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:37 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:37.158+0000 7fe0506d4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-12T15:23:37.426 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:37 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:37.253+0000 7fe0506d4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-12T15:23:37.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.960+0000 7fa658dc0700 1 Processor -- start 2023-07-12T15:23:37.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.960+0000 7fa658dc0700 1 -- start start 2023-07-12T15:23:37.525 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.961+0000 7fa658dc0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 0x7fa654102c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:37.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.961+0000 7fa658dc0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa654077e40 con 0x7fa654102880 2023-07-12T15:23:37.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.961+0000 7fa65259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 0x7fa654102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:37.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.961+0000 7fa65259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 0x7fa654102c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39738/0 (socket says 172.21.15.69:39738) 2023-07-12T15:23:37.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.961+0000 7fa65259c700 1 -- 172.21.15.69:0/1866736000 learned_addr learned my addr 172.21.15.69:0/1866736000 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:37.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.961+0000 7fa65259c700 1 -- 172.21.15.69:0/1866736000 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6541031d0 con 0x7fa654102880 2023-07-12T15:23:37.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.961+0000 7fa65259c700 1 --2- 172.21.15.69:0/1866736000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 0x7fa654102c90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa6480098a0 tx=0x7fa648009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d2484311c595b1 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:37.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.961+0000 7fa65159a700 1 -- 172.21.15.69:0/1866736000 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa648004160 con 0x7fa654102880 2023-07-12T15:23:37.526 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.961+0000 7fa65159a700 1 -- 172.21.15.69:0/1866736000 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fa6480042c0 con 0x7fa654102880 2023-07-12T15:23:37.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.961+0000 7fa65159a700 1 -- 172.21.15.69:0/1866736000 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6480118c0 con 0x7fa654102880 2023-07-12T15:23:37.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.962+0000 7fa658dc0700 1 -- 172.21.15.69:0/1866736000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 msgr2=0x7fa654102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:37.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.962+0000 7fa658dc0700 1 --2- 172.21.15.69:0/1866736000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 0x7fa654102c90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa6480098a0 tx=0x7fa648009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:37.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.962+0000 7fa658dc0700 1 -- 172.21.15.69:0/1866736000 shutdown_connections 2023-07-12T15:23:37.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.962+0000 7fa658dc0700 1 --2- 172.21.15.69:0/1866736000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 0x7fa654102c90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:37.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.962+0000 7fa658dc0700 1 -- 172.21.15.69:0/1866736000 >> 172.21.15.69:0/1866736000 conn(0x7fa6540fe410 msgr2=0x7fa654100880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:37.527 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.962+0000 7fa658dc0700 1 -- 172.21.15.69:0/1866736000 shutdown_connections 2023-07-12T15:23:37.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.962+0000 7fa658dc0700 1 -- 172.21.15.69:0/1866736000 wait complete. 2023-07-12T15:23:37.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.963+0000 7fa658dc0700 1 Processor -- start 2023-07-12T15:23:37.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.963+0000 7fa658dc0700 1 -- start start 2023-07-12T15:23:37.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.963+0000 7fa658dc0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 0x7fa654199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:37.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.963+0000 7fa658dc0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65419a380 con 0x7fa654102880 2023-07-12T15:23:37.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.963+0000 7fa65259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 0x7fa654199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:37.528 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.963+0000 7fa65259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 0x7fa654199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39740/0 (socket says 172.21.15.69:39740) 2023-07-12T15:23:37.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.963+0000 7fa65259c700 1 -- 172.21.15.69:0/476434071 learned_addr learned my addr 172.21.15.69:0/476434071 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:37.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.963+0000 7fa65259c700 1 -- 172.21.15.69:0/476434071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa648009550 con 0x7fa654102880 2023-07-12T15:23:37.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.963+0000 7fa65259c700 1 --2- 172.21.15.69:0/476434071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 0x7fa654199e40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa6480046c0 tx=0x7fa648011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:37.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.964+0000 7fa6437fe700 1 -- 172.21.15.69:0/476434071 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6480118f0 con 0x7fa654102880 2023-07-12T15:23:37.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.964+0000 7fa6437fe700 1 -- 172.21.15.69:0/476434071 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fa64801a430 con 0x7fa654102880 2023-07-12T15:23:37.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.964+0000 7fa658dc0700 1 -- 172.21.15.69:0/476434071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65419a580 con 0x7fa654102880 2023-07-12T15:23:37.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.964+0000 7fa6437fe700 1 -- 172.21.15.69:0/476434071 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6480193c0 con 0x7fa654102880 2023-07-12T15:23:37.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.964+0000 7fa658dc0700 1 -- 172.21.15.69:0/476434071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65419aa20 con 0x7fa654102880 2023-07-12T15:23:37.530 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.964+0000 7fa6437fe700 1 -- 172.21.15.69:0/476434071 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) v1 ==== 45030+0+0 (secure 0 0 0) 0x7fa648019640 con 0x7fa654102880 2023-07-12T15:23:37.530 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.965+0000 7fa658dc0700 1 -- 172.21.15.69:0/476434071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6541936d0 con 0x7fa654102880 2023-07-12T15:23:37.530 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.965+0000 7fa6437fe700 1 --2- 172.21.15.69:0/476434071 >> v2:172.21.15.69:6800/339337620 conn(0x7fa63c038340 0x7fa63c03a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:37.530 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.965+0000 7fa6437fe700 1 -- 172.21.15.69:0/476434071 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fa64804beb0 con 0x7fa654102880 2023-07-12T15:23:37.530 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.965+0000 7fa651d9b700 1 --2- 172.21.15.69:0/476434071 >> v2:172.21.15.69:6800/339337620 conn(0x7fa63c038340 0x7fa63c03a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:37.531 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.965+0000 7fa651d9b700 1 --2- 172.21.15.69:0/476434071 >> v2:172.21.15.69:6800/339337620 conn(0x7fa63c038340 0x7fa63c03a7f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa644006fd0 tx=0x7fa644006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:37.531 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:35.969+0000 7fa6437fe700 1 -- 172.21.15.69:0/476434071 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa64802d440 con 0x7fa654102880 2023-07-12T15:23:37.531 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:36.216+0000 7fa658dc0700 1 -- 172.21.15.69:0/476434071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fa654062130 con 0x7fa654102880 2023-07-12T15:23:37.531 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:36.815+0000 7fa6437fe700 1 -- 172.21.15.69:0/476434071 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 9) v1 ==== 45043+0+0 (secure 0 0 0) 0x7fa648018350 con 0x7fa654102880 2023-07-12T15:23:37.531 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:36.816+0000 7fa6437fe700 1 -- 172.21.15.69:0/476434071 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fa64800bc70 con 0x7fa654102880 2023-07-12T15:23:37.531 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:36.818+0000 7fa658dc0700 1 -- 172.21.15.69:0/476434071 >> v2:172.21.15.69:6800/339337620 conn(0x7fa63c038340 msgr2=0x7fa63c03a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:37.531 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:36.818+0000 7fa658dc0700 1 --2- 172.21.15.69:0/476434071 >> v2:172.21.15.69:6800/339337620 conn(0x7fa63c038340 0x7fa63c03a7f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa644006fd0 tx=0x7fa644006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:37.532 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:36.818+0000 7fa658dc0700 1 -- 172.21.15.69:0/476434071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 msgr2=0x7fa654199e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:37.532 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:36.818+0000 7fa658dc0700 1 --2- 172.21.15.69:0/476434071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 0x7fa654199e40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa6480046c0 tx=0x7fa648011e40 comp rx=0 tx=0).stop 2023-07-12T15:23:37.532 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:36.818+0000 7fa658dc0700 1 -- 172.21.15.69:0/476434071 shutdown_connections 2023-07-12T15:23:37.532 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:36.818+0000 7fa658dc0700 1 --2- 172.21.15.69:0/476434071 >> v2:172.21.15.69:6800/339337620 conn(0x7fa63c038340 0x7fa63c03a7f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:37.532 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:36.818+0000 7fa658dc0700 1 --2- 172.21.15.69:0/476434071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa654102880 0x7fa654199e40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:37.532 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:36.818+0000 7fa658dc0700 1 -- 172.21.15.69:0/476434071 >> 172.21.15.69:0/476434071 conn(0x7fa6540fe410 msgr2=0x7fa65418e440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:37.532 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:36.819+0000 7fa658dc0700 1 -- 172.21.15.69:0/476434071 shutdown_connections 2023-07-12T15:23:37.533 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:36.819+0000 7fa658dc0700 1 -- 172.21.15.69:0/476434071 wait complete. 2023-07-12T15:23:37.696 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:37 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:37.425+0000 7fe0506d4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-12T15:23:37.980 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:37 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:37.750+0000 7fe0506d4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-12T15:23:37.980 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:37 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:37.842+0000 7fe0506d4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-12T15:23:37.981 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/476434071' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-12T15:23:37.981 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:37 smithi069 ceph-mon[94241]: mgrmap e9: y(active, since 15s) 2023-07-12T15:23:38.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:37 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:37.979+0000 7fe0506d4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-12T15:23:38.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:38 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:38.076+0000 7fe0506d4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-12T15:23:38.572 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2023-07-12T15:23:38.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-12T15:23:38.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-12T15:23:38.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "active_name": "y", 2023-07-12T15:23:38.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-12T15:23:38.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2023-07-12T15:23:38.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.961+0000 7f092ab25700 1 Processor -- start 2023-07-12T15:23:38.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.962+0000 7f092ab25700 1 -- start start 2023-07-12T15:23:38.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.962+0000 7f092ab25700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 0x7f0924102c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:38.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.962+0000 7f092ab25700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0924077e40 con 0x7f0924102880 2023-07-12T15:23:38.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.962+0000 7f09288c1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 0x7f0924102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:38.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.962+0000 7f09288c1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 0x7f0924102c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39754/0 (socket says 172.21.15.69:39754) 2023-07-12T15:23:38.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.962+0000 7f09288c1700 1 -- 172.21.15.69:0/3686376123 learned_addr learned my addr 172.21.15.69:0/3686376123 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:38.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.962+0000 7f09288c1700 1 -- 172.21.15.69:0/3686376123 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09241031d0 con 0x7f0924102880 2023-07-12T15:23:38.575 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.962+0000 7f09288c1700 1 --2- 172.21.15.69:0/3686376123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 0x7f0924102c90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f09180098a0 tx=0x7f0918009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26605220e1df098e server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:38.575 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.962+0000 7f09237fe700 1 -- 172.21.15.69:0/3686376123 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0918004160 con 0x7f0924102880 2023-07-12T15:23:38.575 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.963+0000 7f09237fe700 1 -- 172.21.15.69:0/3686376123 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f09180042c0 con 0x7f0924102880 2023-07-12T15:23:38.575 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.963+0000 7f09237fe700 1 -- 172.21.15.69:0/3686376123 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f09180118c0 con 0x7f0924102880 2023-07-12T15:23:38.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.963+0000 7f092ab25700 1 -- 172.21.15.69:0/3686376123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 msgr2=0x7f0924102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:38.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.963+0000 7f092ab25700 1 --2- 172.21.15.69:0/3686376123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 0x7f0924102c90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f09180098a0 tx=0x7f0918009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:38.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.963+0000 7f092ab25700 1 -- 172.21.15.69:0/3686376123 shutdown_connections 2023-07-12T15:23:38.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.963+0000 7f092ab25700 1 --2- 172.21.15.69:0/3686376123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 0x7f0924102c90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:38.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.963+0000 7f092ab25700 1 -- 172.21.15.69:0/3686376123 >> 172.21.15.69:0/3686376123 conn(0x7f09240fe410 msgr2=0x7f0924100880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:38.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.963+0000 7f092ab25700 1 -- 172.21.15.69:0/3686376123 shutdown_connections 2023-07-12T15:23:38.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.963+0000 7f092ab25700 1 -- 172.21.15.69:0/3686376123 wait complete. 2023-07-12T15:23:38.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.964+0000 7f092ab25700 1 Processor -- start 2023-07-12T15:23:38.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.964+0000 7f092ab25700 1 -- start start 2023-07-12T15:23:38.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.964+0000 7f092ab25700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 0x7f0924199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:38.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.964+0000 7f092ab25700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092419a360 con 0x7f0924102880 2023-07-12T15:23:38.578 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.964+0000 7f09288c1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 0x7f0924199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:38.578 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.964+0000 7f09288c1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 0x7f0924199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39756/0 (socket says 172.21.15.69:39756) 2023-07-12T15:23:38.578 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.964+0000 7f09288c1700 1 -- 172.21.15.69:0/1691457883 learned_addr learned my addr 172.21.15.69:0/1691457883 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:38.578 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.965+0000 7f09288c1700 1 -- 172.21.15.69:0/1691457883 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0918009550 con 0x7f0924102880 2023-07-12T15:23:38.578 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.965+0000 7f09288c1700 1 --2- 172.21.15.69:0/1691457883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 0x7f0924199e20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f09180046c0 tx=0x7f0918011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:38.578 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.965+0000 7f0921ffb700 1 -- 172.21.15.69:0/1691457883 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f09180118f0 con 0x7f0924102880 2023-07-12T15:23:38.578 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.965+0000 7f0921ffb700 1 -- 172.21.15.69:0/1691457883 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f091801a430 con 0x7f0924102880 2023-07-12T15:23:38.579 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.965+0000 7f092ab25700 1 -- 172.21.15.69:0/1691457883 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092419a560 con 0x7f0924102880 2023-07-12T15:23:38.579 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.965+0000 7f0921ffb700 1 -- 172.21.15.69:0/1691457883 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f09180193c0 con 0x7f0924102880 2023-07-12T15:23:38.579 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.965+0000 7f092ab25700 1 -- 172.21.15.69:0/1691457883 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092419aa00 con 0x7f0924102880 2023-07-12T15:23:38.579 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.966+0000 7f0921ffb700 1 -- 172.21.15.69:0/1691457883 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 9) v1 ==== 45043+0+0 (secure 0 0 0) 0x7f09180036a0 con 0x7f0924102880 2023-07-12T15:23:38.579 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.966+0000 7f0921ffb700 1 --2- 172.21.15.69:0/1691457883 >> v2:172.21.15.69:6800/339337620 conn(0x7f0914038390 0x7f091403a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:38.579 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.966+0000 7f0923fff700 1 -- 172.21.15.69:0/1691457883 >> v2:172.21.15.69:6800/339337620 conn(0x7f0914038390 msgr2=0x7f091403a840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/339337620 2023-07-12T15:23:38.579 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.966+0000 7f0921ffb700 1 -- 172.21.15.69:0/1691457883 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f091804bd10 con 0x7f0924102880 2023-07-12T15:23:38.579 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.966+0000 7f0923fff700 1 --2- 172.21.15.69:0/1691457883 >> v2:172.21.15.69:6800/339337620 conn(0x7f0914038390 0x7f091403a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-12T15:23:38.580 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.966+0000 7f092ab25700 1 -- 172.21.15.69:0/1691457883 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09241936b0 con 0x7f0924102880 2023-07-12T15:23:38.580 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:37.971+0000 7f0921ffb700 1 -- 172.21.15.69:0/1691457883 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0918019640 con 0x7f0924102880 2023-07-12T15:23:38.580 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.167+0000 7f0923fff700 1 -- 172.21.15.69:0/1691457883 >> v2:172.21.15.69:6800/339337620 conn(0x7f0914038390 msgr2=0x7f091403a840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/339337620 2023-07-12T15:23:38.580 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.167+0000 7f0923fff700 1 --2- 172.21.15.69:0/1691457883 >> v2:172.21.15.69:6800/339337620 conn(0x7f0914038390 0x7f091403a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-12T15:23:38.580 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.218+0000 7f092ab25700 1 -- 172.21.15.69:0/1691457883 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f092402cd90 con 0x7f0924102880 2023-07-12T15:23:38.581 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.218+0000 7f0921ffb700 1 -- 172.21.15.69:0/1691457883 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f0918023b40 con 0x7f0924102880 2023-07-12T15:23:38.581 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.220+0000 7f092ab25700 1 -- 172.21.15.69:0/1691457883 >> v2:172.21.15.69:6800/339337620 conn(0x7f0914038390 msgr2=0x7f091403a840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-12T15:23:38.581 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.220+0000 7f092ab25700 1 --2- 172.21.15.69:0/1691457883 >> v2:172.21.15.69:6800/339337620 conn(0x7f0914038390 0x7f091403a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:38.581 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.220+0000 7f092ab25700 1 -- 172.21.15.69:0/1691457883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 msgr2=0x7f0924199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:38.581 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.220+0000 7f092ab25700 1 --2- 172.21.15.69:0/1691457883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 0x7f0924199e20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f09180046c0 tx=0x7f0918011e40 comp rx=0 tx=0).stop 2023-07-12T15:23:38.581 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.220+0000 7f092ab25700 1 -- 172.21.15.69:0/1691457883 shutdown_connections 2023-07-12T15:23:38.581 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.220+0000 7f092ab25700 1 --2- 172.21.15.69:0/1691457883 >> v2:172.21.15.69:6800/339337620 conn(0x7f0914038390 0x7f091403a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:38.582 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.220+0000 7f092ab25700 1 --2- 172.21.15.69:0/1691457883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0924102880 0x7f0924199e20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:38.582 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.220+0000 7f092ab25700 1 -- 172.21.15.69:0/1691457883 >> 172.21.15.69:0/1691457883 conn(0x7f09240fe410 msgr2=0x7f092418e440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:38.582 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.221+0000 7f092ab25700 1 -- 172.21.15.69:0/1691457883 shutdown_connections 2023-07-12T15:23:38.582 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:38.221+0000 7f092ab25700 1 -- 172.21.15.69:0/1691457883 wait complete. 2023-07-12T15:23:38.582 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for the mgr to restart... 2023-07-12T15:23:38.582 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mgr epoch 9... 2023-07-12T15:23:38.780 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:38 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:38.515+0000 7fe0506d4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-12T15:23:39.110 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1691457883' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-12T15:23:39.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:39.108+0000 7fe0506d4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-12T15:23:39.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:39.277+0000 7fe0506d4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-12T15:23:40.293 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:40 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:40.029+0000 7fe0506d4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-12T15:23:40.293 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:40 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:40.119+0000 7fe0506d4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-12T15:23:40.573 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:40 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:40.292+0000 7fe0506d4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-12T15:23:40.960 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:40 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:40.571+0000 7fe0506d4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-12T15:23:40.961 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:40 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:40.812+0000 7fe0506d4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-12T15:23:41.219 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:40 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:40.959+0000 7fe0506d4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-12T15:23:41.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:41 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:41.218+0000 7fe0506d4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-12T15:23:42.232 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:41 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:41.909+0000 7fe0506d4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-12T15:23:42.232 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:42 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:42.040+0000 7fe0506d4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-12T15:23:42.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:42 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:42.230+0000 7fe0506d4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-12T15:23:42.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:42 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:42.344+0000 7fe0506d4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-12T15:23:42.985 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:42 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:42.559+0000 7fe0506d4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-12T15:23:43.735 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:23:43 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:23:43.410+0000 7fe0506d4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-12T15:23:43.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:43 smithi069 ceph-mon[94241]: Active manager daemon y restarted 2023-07-12T15:23:43.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:43 smithi069 ceph-mon[94241]: Activating manager daemon y 2023-07-12T15:23:43.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:43 smithi069 ceph-mon[94241]: osdmap e3: 0 total, 0 up, 0 in 2023-07-12T15:23:43.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:43 smithi069 ceph-mon[94241]: mgrmap e10: y(active, starting, since 0.00310174s) 2023-07-12T15:23:43.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:43 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:23:43.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:43 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-12T15:23:43.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:43 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-12T15:23:43.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:43 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-12T15:23:43.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:43 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-12T15:23:43.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:43 smithi069 ceph-mon[94241]: Manager daemon y is now available 2023-07-12T15:23:43.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:43 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:23:43.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:43 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:23:44.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-12T15:23:44.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-12T15:23:44.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:44 smithi069 ceph-mon[94241]: mgrmap e11: y(active, since 1.00533s) 2023-07-12T15:23:45.335 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2023-07-12T15:23:45.337 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-12T15:23:45.337 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-12T15:23:45.338 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2023-07-12T15:23:45.338 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.075+0000 7fef5d065700 1 Processor -- start 2023-07-12T15:23:45.338 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.075+0000 7fef5d065700 1 -- start start 2023-07-12T15:23:45.338 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.075+0000 7fef5d065700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 0x7fef58102c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:45.338 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.075+0000 7fef5d065700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef58077e60 con 0x7fef58102860 2023-07-12T15:23:45.338 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.075+0000 7fef56d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 0x7fef58102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:45.338 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.075+0000 7fef56d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 0x7fef58102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39764/0 (socket says 172.21.15.69:39764) 2023-07-12T15:23:45.339 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.075+0000 7fef56d9d700 1 -- 172.21.15.69:0/68658425 learned_addr learned my addr 172.21.15.69:0/68658425 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:45.339 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.076+0000 7fef56d9d700 1 -- 172.21.15.69:0/68658425 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef581031b0 con 0x7fef58102860 2023-07-12T15:23:45.339 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.076+0000 7fef56d9d700 1 --2- 172.21.15.69:0/68658425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 0x7fef58102c70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fef480098a0 tx=0x7fef48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89fb30b5ec8b8fb0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:45.343 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.076+0000 7fef55d9b700 1 -- 172.21.15.69:0/68658425 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef48004160 con 0x7fef58102860 2023-07-12T15:23:45.344 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.076+0000 7fef55d9b700 1 -- 172.21.15.69:0/68658425 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fef480042c0 con 0x7fef58102860 2023-07-12T15:23:45.344 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.076+0000 7fef55d9b700 1 -- 172.21.15.69:0/68658425 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef480118c0 con 0x7fef58102860 2023-07-12T15:23:45.344 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.076+0000 7fef5d065700 1 -- 172.21.15.69:0/68658425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 msgr2=0x7fef58102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:45.344 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.076+0000 7fef5d065700 1 --2- 172.21.15.69:0/68658425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 0x7fef58102c70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fef480098a0 tx=0x7fef48009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:45.344 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.077+0000 7fef5d065700 1 -- 172.21.15.69:0/68658425 shutdown_connections 2023-07-12T15:23:45.344 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.077+0000 7fef5d065700 1 --2- 172.21.15.69:0/68658425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 0x7fef58102c70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:45.345 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.077+0000 7fef5d065700 1 -- 172.21.15.69:0/68658425 >> 172.21.15.69:0/68658425 conn(0x7fef580fe410 msgr2=0x7fef58100880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:45.345 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.077+0000 7fef5d065700 1 -- 172.21.15.69:0/68658425 shutdown_connections 2023-07-12T15:23:45.345 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.077+0000 7fef5d065700 1 -- 172.21.15.69:0/68658425 wait complete. 2023-07-12T15:23:45.345 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.077+0000 7fef5d065700 1 Processor -- start 2023-07-12T15:23:45.345 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.078+0000 7fef5d065700 1 -- start start 2023-07-12T15:23:45.345 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.078+0000 7fef5d065700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 0x7fef58199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:45.345 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.078+0000 7fef5d065700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5819a350 con 0x7fef58102860 2023-07-12T15:23:45.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.078+0000 7fef56d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 0x7fef58199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:45.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.078+0000 7fef56d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 0x7fef58199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39766/0 (socket says 172.21.15.69:39766) 2023-07-12T15:23:45.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.078+0000 7fef56d9d700 1 -- 172.21.15.69:0/1642940994 learned_addr learned my addr 172.21.15.69:0/1642940994 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:45.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.078+0000 7fef56d9d700 1 -- 172.21.15.69:0/1642940994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef48009550 con 0x7fef58102860 2023-07-12T15:23:45.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.078+0000 7fef56d9d700 1 --2- 172.21.15.69:0/1642940994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 0x7fef58199e10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fef480046c0 tx=0x7fef48011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:45.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.078+0000 7fef3ffff700 1 -- 172.21.15.69:0/1642940994 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef480118f0 con 0x7fef58102860 2023-07-12T15:23:45.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.078+0000 7fef5d065700 1 -- 172.21.15.69:0/1642940994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5819a550 con 0x7fef58102860 2023-07-12T15:23:45.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.078+0000 7fef5d065700 1 -- 172.21.15.69:0/1642940994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5819a9f0 con 0x7fef58102860 2023-07-12T15:23:45.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.078+0000 7fef3ffff700 1 -- 172.21.15.69:0/1642940994 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fef4801a430 con 0x7fef58102860 2023-07-12T15:23:45.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.079+0000 7fef3ffff700 1 -- 172.21.15.69:0/1642940994 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef480193c0 con 0x7fef58102860 2023-07-12T15:23:45.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.079+0000 7fef3ffff700 1 -- 172.21.15.69:0/1642940994 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 9) v1 ==== 45043+0+0 (secure 0 0 0) 0x7fef480195f0 con 0x7fef58102860 2023-07-12T15:23:45.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.079+0000 7fef3ffff700 1 --2- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 0x7fef4003a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:45.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.079+0000 7fef5d065700 1 -- 172.21.15.69:0/1642940994 --> v2:172.21.15.69:6800/339337620 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef44000d40 con 0x7fef400383e0 2023-07-12T15:23:45.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.079+0000 7fef3ffff700 1 -- 172.21.15.69:0/1642940994 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fef4804cb70 con 0x7fef58102860 2023-07-12T15:23:45.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.079+0000 7fef5659c700 1 -- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 msgr2=0x7fef4003a890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/339337620 2023-07-12T15:23:45.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.079+0000 7fef5659c700 1 --2- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 0x7fef4003a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-12T15:23:45.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.280+0000 7fef5659c700 1 -- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 msgr2=0x7fef4003a890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/339337620 2023-07-12T15:23:45.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.280+0000 7fef5659c700 1 --2- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 0x7fef4003a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-12T15:23:45.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.681+0000 7fef5659c700 1 -- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 msgr2=0x7fef4003a890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/339337620 2023-07-12T15:23:45.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:39.681+0000 7fef5659c700 1 --2- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 0x7fef4003a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-12T15:23:45.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:40.482+0000 7fef5659c700 1 -- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 msgr2=0x7fef4003a890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/339337620 2023-07-12T15:23:45.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:40.482+0000 7fef5659c700 1 --2- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 0x7fef4003a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-12T15:23:45.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:42.083+0000 7fef5659c700 1 -- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 msgr2=0x7fef4003a890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/339337620 2023-07-12T15:23:45.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:42.083+0000 7fef5659c700 1 --2- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 0x7fef4003a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-12T15:23:45.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:43.425+0000 7fef3ffff700 1 -- 172.21.15.69:0/1642940994 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mgrmap(e 10) v1 ==== 44845+0+0 (secure 0 0 0) 0x7fef4802d770 con 0x7fef58102860 2023-07-12T15:23:45.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:43.425+0000 7fef3ffff700 1 -- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 msgr2=0x7fef4003a890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-12T15:23:45.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:43.425+0000 7fef3ffff700 1 --2- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 0x7fef4003a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:45.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.426+0000 7fef3ffff700 1 -- 172.21.15.69:0/1642940994 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 11) v1 ==== 44937+0+0 (secure 0 0 0) 0x7fef4804d8e0 con 0x7fef58102860 2023-07-12T15:23:45.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.426+0000 7fef3ffff700 1 --2- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/284433772 conn(0x7fef4003bc50 0x7fef4003e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:45.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.426+0000 7fef3ffff700 1 -- 172.21.15.69:0/1642940994 --> v2:172.21.15.69:6800/284433772 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef44000d40 con 0x7fef4003bc50 2023-07-12T15:23:45.350 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.427+0000 7fef5659c700 1 --2- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/284433772 conn(0x7fef4003bc50 0x7fef4003e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:45.350 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.427+0000 7fef5659c700 1 --2- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/284433772 conn(0x7fef4003bc50 0x7fef4003e030 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fef4c003a10 tx=0x7fef4c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:45.350 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.428+0000 7fef3ffff700 1 -- 172.21.15.69:0/1642940994 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fef44000d40 con 0x7fef4003bc50 2023-07-12T15:23:45.350 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.434+0000 7fef5d065700 1 -- 172.21.15.69:0/1642940994 --> v2:172.21.15.69:6800/284433772 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fef44002800 con 0x7fef4003bc50 2023-07-12T15:23:45.350 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.434+0000 7fef3ffff700 1 -- 172.21.15.69:0/1642940994 <== mgr.14152 v2:172.21.15.69:6800/284433772 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fef44002800 con 0x7fef4003bc50 2023-07-12T15:23:45.350 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.435+0000 7fef5d065700 1 -- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/284433772 conn(0x7fef4003bc50 msgr2=0x7fef4003e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:45.350 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.435+0000 7fef5d065700 1 --2- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/284433772 conn(0x7fef4003bc50 0x7fef4003e030 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fef4c003a10 tx=0x7fef4c009270 comp rx=0 tx=0).stop 2023-07-12T15:23:45.351 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.435+0000 7fef5d065700 1 -- 172.21.15.69:0/1642940994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 msgr2=0x7fef58199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:45.351 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.435+0000 7fef5d065700 1 --2- 172.21.15.69:0/1642940994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 0x7fef58199e10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fef480046c0 tx=0x7fef48011e40 comp rx=0 tx=0).stop 2023-07-12T15:23:45.351 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.435+0000 7fef5d065700 1 -- 172.21.15.69:0/1642940994 shutdown_connections 2023-07-12T15:23:45.351 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.435+0000 7fef5d065700 1 --2- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/284433772 conn(0x7fef4003bc50 0x7fef4003e030 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:45.351 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.435+0000 7fef5d065700 1 --2- 172.21.15.69:0/1642940994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef58102860 0x7fef58199e10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:45.351 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.435+0000 7fef5d065700 1 --2- 172.21.15.69:0/1642940994 >> v2:172.21.15.69:6800/339337620 conn(0x7fef400383e0 0x7fef4003a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:45.351 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.435+0000 7fef5d065700 1 -- 172.21.15.69:0/1642940994 >> 172.21.15.69:0/1642940994 conn(0x7fef580fe410 msgr2=0x7fef5818e420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:45.351 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.435+0000 7fef5d065700 1 -- 172.21.15.69:0/1642940994 shutdown_connections 2023-07-12T15:23:45.352 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:44.435+0000 7fef5d065700 1 -- 172.21.15.69:0/1642940994 wait complete. 2023-07-12T15:23:45.352 INFO:teuthology.orchestra.run.smithi069.stdout:mgr epoch 9 is available 2023-07-12T15:23:45.352 INFO:teuthology.orchestra.run.smithi069.stdout:Generating a dashboard self-signed certificate... 2023-07-12T15:23:45.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:45 smithi069 ceph-mon[94241]: [12/Jul/2023:15:23:44] ENGINE Bus STARTING 2023-07-12T15:23:45.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:45 smithi069 ceph-mon[94241]: [12/Jul/2023:15:23:44] ENGINE Serving on https://172.21.15.69:7150 2023-07-12T15:23:45.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:45 smithi069 ceph-mon[94241]: [12/Jul/2023:15:23:44] ENGINE Serving on http://172.21.15.69:8765 2023-07-12T15:23:45.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:45 smithi069 ceph-mon[94241]: [12/Jul/2023:15:23:44] ENGINE Bus STARTED 2023-07-12T15:23:46.650 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-12T15:23:46.650 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.972+0000 7ff9fd89e700 1 Processor -- start 2023-07-12T15:23:46.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.972+0000 7ff9fd89e700 1 -- start start 2023-07-12T15:23:46.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.972+0000 7ff9fd89e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 0x7ff9f81043d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:46.651 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.972+0000 7ff9fd89e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f8104910 con 0x7ff9f8103fc0 2023-07-12T15:23:46.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.972+0000 7ff9f6ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 0x7ff9f81043d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:46.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.972+0000 7ff9f6ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 0x7ff9f81043d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39822/0 (socket says 172.21.15.69:39822) 2023-07-12T15:23:46.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.972+0000 7ff9f6ffd700 1 -- 172.21.15.69:0/1747995177 learned_addr learned my addr 172.21.15.69:0/1747995177 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:46.652 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.972+0000 7ff9f6ffd700 1 -- 172.21.15.69:0/1747995177 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f8104a50 con 0x7ff9f8103fc0 2023-07-12T15:23:46.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.973+0000 7ff9f6ffd700 1 --2- 172.21.15.69:0/1747995177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 0x7ff9f81043d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80098a0 tx=0x7ff9e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6084eab9eabec1b server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:46.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.973+0000 7ff9f5ffb700 1 -- 172.21.15.69:0/1747995177 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9e8004160 con 0x7ff9f8103fc0 2023-07-12T15:23:46.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.973+0000 7ff9f5ffb700 1 -- 172.21.15.69:0/1747995177 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7ff9e80042c0 con 0x7ff9f8103fc0 2023-07-12T15:23:46.653 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.973+0000 7ff9f5ffb700 1 -- 172.21.15.69:0/1747995177 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9e80118c0 con 0x7ff9f8103fc0 2023-07-12T15:23:46.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.974+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1747995177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 msgr2=0x7ff9f81043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:46.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.974+0000 7ff9fd89e700 1 --2- 172.21.15.69:0/1747995177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 0x7ff9f81043d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80098a0 tx=0x7ff9e8009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:46.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.974+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1747995177 shutdown_connections 2023-07-12T15:23:46.654 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.974+0000 7ff9fd89e700 1 --2- 172.21.15.69:0/1747995177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 0x7ff9f81043d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:46.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.974+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1747995177 >> 172.21.15.69:0/1747995177 conn(0x7ff9f80ff630 msgr2=0x7ff9f8101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:46.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.974+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1747995177 shutdown_connections 2023-07-12T15:23:46.655 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.974+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1747995177 wait complete. 2023-07-12T15:23:46.656 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.974+0000 7ff9fd89e700 1 Processor -- start 2023-07-12T15:23:46.656 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.975+0000 7ff9fd89e700 1 -- start start 2023-07-12T15:23:46.656 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.975+0000 7ff9fd89e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 0x7ff9f8197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:46.656 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.975+0000 7ff9fd89e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f81980b0 con 0x7ff9f8103fc0 2023-07-12T15:23:46.657 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.975+0000 7ff9f6ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 0x7ff9f8197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:46.657 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.975+0000 7ff9f6ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 0x7ff9f8197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39824/0 (socket says 172.21.15.69:39824) 2023-07-12T15:23:46.657 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.975+0000 7ff9f6ffd700 1 -- 172.21.15.69:0/1006261872 learned_addr learned my addr 172.21.15.69:0/1006261872 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:46.658 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.975+0000 7ff9f6ffd700 1 -- 172.21.15.69:0/1006261872 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e8009550 con 0x7ff9f8103fc0 2023-07-12T15:23:46.658 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.976+0000 7ff9f6ffd700 1 --2- 172.21.15.69:0/1006261872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 0x7ff9f8197b70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80046c0 tx=0x7ff9e80119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:46.658 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.976+0000 7ff9fc89c700 1 -- 172.21.15.69:0/1006261872 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9e8011e20 con 0x7ff9f8103fc0 2023-07-12T15:23:46.658 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.976+0000 7ff9fc89c700 1 -- 172.21.15.69:0/1006261872 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7ff9e80038b0 con 0x7ff9f8103fc0 2023-07-12T15:23:46.658 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.976+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1006261872 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f81982b0 con 0x7ff9f8103fc0 2023-07-12T15:23:46.659 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.976+0000 7ff9fc89c700 1 -- 172.21.15.69:0/1006261872 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9e80193c0 con 0x7ff9f8103fc0 2023-07-12T15:23:46.659 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.976+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1006261872 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f8198750 con 0x7ff9f8103fc0 2023-07-12T15:23:46.659 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.976+0000 7ff9fc89c700 1 -- 172.21.15.69:0/1006261872 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 12) v1 ==== 45043+0+0 (secure 0 0 0) 0x7ff9e8019680 con 0x7ff9f8103fc0 2023-07-12T15:23:46.659 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.977+0000 7ff9fc89c700 1 --2- 172.21.15.69:0/1006261872 >> v2:172.21.15.69:6800/284433772 conn(0x7ff9e00383d0 0x7ff9e003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:46.660 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.977+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1006261872 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f8062130 con 0x7ff9f8103fc0 2023-07-12T15:23:46.660 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.977+0000 7ff9fc89c700 1 -- 172.21.15.69:0/1006261872 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7ff9e804d0c0 con 0x7ff9f8103fc0 2023-07-12T15:23:46.660 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.977+0000 7ff9f67fc700 1 --2- 172.21.15.69:0/1006261872 >> v2:172.21.15.69:6800/284433772 conn(0x7ff9e00383d0 0x7ff9e003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:46.661 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.977+0000 7ff9f67fc700 1 --2- 172.21.15.69:0/1006261872 >> v2:172.21.15.69:6800/284433772 conn(0x7ff9e00383d0 0x7ff9e003a880 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec006fd0 tx=0x7ff9ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:46.661 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:45.982+0000 7ff9fc89c700 1 -- 172.21.15.69:0/1006261872 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff9e8019980 con 0x7ff9f8103fc0 2023-07-12T15:23:46.661 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:46.191+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1006261872 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9f8100980 con 0x7ff9e00383d0 2023-07-12T15:23:46.662 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:46.273+0000 7ff9fc89c700 1 -- 172.21.15.69:0/1006261872 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7ff9f8100980 con 0x7ff9e00383d0 2023-07-12T15:23:46.662 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:46.276+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1006261872 >> v2:172.21.15.69:6800/284433772 conn(0x7ff9e00383d0 msgr2=0x7ff9e003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:46.662 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:46.276+0000 7ff9fd89e700 1 --2- 172.21.15.69:0/1006261872 >> v2:172.21.15.69:6800/284433772 conn(0x7ff9e00383d0 0x7ff9e003a880 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec006fd0 tx=0x7ff9ec006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:46.662 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:46.276+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1006261872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 msgr2=0x7ff9f8197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:46.663 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:46.276+0000 7ff9fd89e700 1 --2- 172.21.15.69:0/1006261872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 0x7ff9f8197b70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80046c0 tx=0x7ff9e80119d0 comp rx=0 tx=0).stop 2023-07-12T15:23:46.663 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:46.276+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1006261872 shutdown_connections 2023-07-12T15:23:46.663 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:46.276+0000 7ff9fd89e700 1 --2- 172.21.15.69:0/1006261872 >> v2:172.21.15.69:6800/284433772 conn(0x7ff9e00383d0 0x7ff9e003a880 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:46.664 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:46.276+0000 7ff9fd89e700 1 --2- 172.21.15.69:0/1006261872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8103fc0 0x7ff9f8197b70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:46.664 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:46.276+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1006261872 >> 172.21.15.69:0/1006261872 conn(0x7ff9f80ff630 msgr2=0x7ff9f8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:46.664 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:46.276+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1006261872 shutdown_connections 2023-07-12T15:23:46.665 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:46.276+0000 7ff9fd89e700 1 -- 172.21.15.69:0/1006261872 wait complete. 2023-07-12T15:23:46.665 INFO:teuthology.orchestra.run.smithi069.stdout:Creating initial admin user... 2023-07-12T15:23:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:46 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:46 smithi069 ceph-mon[94241]: mgrmap e12: y(active, since 2s) 2023-07-12T15:23:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:46 smithi069 ceph-mon[94241]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:23:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:46 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:46 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:47.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:46 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:48.397 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$mkTx8MG..po1b.JqfBvWMOCW299Wnc0VxRMuY0.eGzWeJIJ0S5PjK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689175427, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-12T15:23:48.397 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.188+0000 7f2fe3723700 1 Processor -- start 2023-07-12T15:23:48.397 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.188+0000 7f2fe3723700 1 -- start start 2023-07-12T15:23:48.397 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.188+0000 7f2fe3723700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 0x7f2fdc106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:48.398 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.188+0000 7f2fe3723700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc077d90 con 0x7f2fdc104ad0 2023-07-12T15:23:48.398 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.188+0000 7f2fe14bf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 0x7f2fdc106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:48.398 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.189+0000 7f2fe14bf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 0x7f2fdc106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39828/0 (socket says 172.21.15.69:39828) 2023-07-12T15:23:48.398 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.189+0000 7f2fe14bf700 1 -- 172.21.15.69:0/2924115192 learned_addr learned my addr 172.21.15.69:0/2924115192 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:48.398 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.189+0000 7f2fe14bf700 1 -- 172.21.15.69:0/2924115192 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc077ed0 con 0x7f2fdc104ad0 2023-07-12T15:23:48.398 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.189+0000 7f2fe14bf700 1 --2- 172.21.15.69:0/2924115192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 0x7f2fdc106eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80098a0 tx=0x7f2fd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a208766b2b0daf53 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:48.398 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.189+0000 7f2fd3fff700 1 -- 172.21.15.69:0/2924115192 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fd8004160 con 0x7f2fdc104ad0 2023-07-12T15:23:48.399 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.189+0000 7f2fd3fff700 1 -- 172.21.15.69:0/2924115192 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f2fd80042c0 con 0x7f2fdc104ad0 2023-07-12T15:23:48.399 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.189+0000 7f2fd3fff700 1 -- 172.21.15.69:0/2924115192 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fd80118c0 con 0x7f2fdc104ad0 2023-07-12T15:23:48.399 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.190+0000 7f2fe3723700 1 -- 172.21.15.69:0/2924115192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 msgr2=0x7f2fdc106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:48.399 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.190+0000 7f2fe3723700 1 --2- 172.21.15.69:0/2924115192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 0x7f2fdc106eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80098a0 tx=0x7f2fd8009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:48.399 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.190+0000 7f2fe3723700 1 -- 172.21.15.69:0/2924115192 shutdown_connections 2023-07-12T15:23:48.399 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.190+0000 7f2fe3723700 1 --2- 172.21.15.69:0/2924115192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 0x7f2fdc106eb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:48.399 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.190+0000 7f2fe3723700 1 -- 172.21.15.69:0/2924115192 >> 172.21.15.69:0/2924115192 conn(0x7f2fdc0fe4d0 msgr2=0x7f2fdc100940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:48.400 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.190+0000 7f2fe3723700 1 -- 172.21.15.69:0/2924115192 shutdown_connections 2023-07-12T15:23:48.400 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.190+0000 7f2fe3723700 1 -- 172.21.15.69:0/2924115192 wait complete. 2023-07-12T15:23:48.400 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.190+0000 7f2fe3723700 1 Processor -- start 2023-07-12T15:23:48.400 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.191+0000 7f2fe3723700 1 -- start start 2023-07-12T15:23:48.400 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.191+0000 7f2fe3723700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 0x7f2fdc197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:48.400 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.191+0000 7f2fe3723700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc198190 con 0x7f2fdc104ad0 2023-07-12T15:23:48.400 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.191+0000 7f2fe14bf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 0x7f2fdc197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:48.401 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.191+0000 7f2fe14bf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 0x7f2fdc197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39830/0 (socket says 172.21.15.69:39830) 2023-07-12T15:23:48.401 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.191+0000 7f2fe14bf700 1 -- 172.21.15.69:0/1116123454 learned_addr learned my addr 172.21.15.69:0/1116123454 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:48.401 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.191+0000 7f2fe14bf700 1 -- 172.21.15.69:0/1116123454 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd8009550 con 0x7f2fdc104ad0 2023-07-12T15:23:48.401 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.191+0000 7f2fe14bf700 1 --2- 172.21.15.69:0/1116123454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 0x7f2fdc197c50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80046c0 tx=0x7f2fd8011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:48.401 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.191+0000 7f2fd27fc700 1 -- 172.21.15.69:0/1116123454 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fd80036a0 con 0x7f2fdc104ad0 2023-07-12T15:23:48.401 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.191+0000 7f2fe3723700 1 -- 172.21.15.69:0/1116123454 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fdc198390 con 0x7f2fdc104ad0 2023-07-12T15:23:48.401 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.191+0000 7f2fd27fc700 1 -- 172.21.15.69:0/1116123454 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f2fd8003800 con 0x7f2fdc104ad0 2023-07-12T15:23:48.401 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.191+0000 7f2fd27fc700 1 -- 172.21.15.69:0/1116123454 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fd8024e70 con 0x7f2fdc104ad0 2023-07-12T15:23:48.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.192+0000 7f2fe3723700 1 -- 172.21.15.69:0/1116123454 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fdc198830 con 0x7f2fdc104ad0 2023-07-12T15:23:48.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.192+0000 7f2fd27fc700 1 -- 172.21.15.69:0/1116123454 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 12) v1 ==== 45043+0+0 (secure 0 0 0) 0x7f2fd801a480 con 0x7f2fdc104ad0 2023-07-12T15:23:48.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.192+0000 7f2fe3723700 1 -- 172.21.15.69:0/1116123454 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fdc1080a0 con 0x7f2fdc104ad0 2023-07-12T15:23:48.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.193+0000 7f2fd27fc700 1 --2- 172.21.15.69:0/1116123454 >> v2:172.21.15.69:6800/284433772 conn(0x7f2fc8038370 0x7f2fc803a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:48.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.193+0000 7f2fd27fc700 1 -- 172.21.15.69:0/1116123454 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f2fd804c140 con 0x7f2fdc104ad0 2023-07-12T15:23:48.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.193+0000 7f2fe0cbe700 1 --2- 172.21.15.69:0/1116123454 >> v2:172.21.15.69:6800/284433772 conn(0x7f2fc8038370 0x7f2fc803a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:48.402 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.193+0000 7f2fe0cbe700 1 --2- 172.21.15.69:0/1116123454 >> v2:172.21.15.69:6800/284433772 conn(0x7f2fc8038370 0x7f2fc803a820 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc006fd0 tx=0x7f2fcc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.197+0000 7f2fd27fc700 1 -- 172.21.15.69:0/1116123454 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2fd8018940 con 0x7f2fdc104ad0 2023-07-12T15:23:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.417+0000 7f2fe3723700 1 -- 172.21.15.69:0/1116123454 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f2fdc061f50 con 0x7f2fc8038370 2023-07-12T15:23:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.642+0000 7f2fd27fc700 1 -- 172.21.15.69:0/1116123454 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f2fdc061f50 con 0x7f2fc8038370 2023-07-12T15:23:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.645+0000 7f2fe3723700 1 -- 172.21.15.69:0/1116123454 >> v2:172.21.15.69:6800/284433772 conn(0x7f2fc8038370 msgr2=0x7f2fc803a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.645+0000 7f2fe3723700 1 --2- 172.21.15.69:0/1116123454 >> v2:172.21.15.69:6800/284433772 conn(0x7f2fc8038370 0x7f2fc803a820 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc006fd0 tx=0x7f2fcc006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.645+0000 7f2fe3723700 1 -- 172.21.15.69:0/1116123454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 msgr2=0x7f2fdc197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.645+0000 7f2fe3723700 1 --2- 172.21.15.69:0/1116123454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 0x7f2fdc197c50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80046c0 tx=0x7f2fd8011920 comp rx=0 tx=0).stop 2023-07-12T15:23:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.645+0000 7f2fe3723700 1 -- 172.21.15.69:0/1116123454 shutdown_connections 2023-07-12T15:23:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.645+0000 7f2fe3723700 1 --2- 172.21.15.69:0/1116123454 >> v2:172.21.15.69:6800/284433772 conn(0x7f2fc8038370 0x7f2fc803a820 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.645+0000 7f2fe3723700 1 --2- 172.21.15.69:0/1116123454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fdc104ad0 0x7f2fdc197c50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.645+0000 7f2fe3723700 1 -- 172.21.15.69:0/1116123454 >> 172.21.15.69:0/1116123454 conn(0x7f2fdc0fe4d0 msgr2=0x7f2fdc0ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.645+0000 7f2fe3723700 1 -- 172.21.15.69:0/1116123454 shutdown_connections 2023-07-12T15:23:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:47.645+0000 7f2fe3723700 1 -- 172.21.15.69:0/1116123454 wait complete. 2023-07-12T15:23:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:Fetching dashboard port number... 2023-07-12T15:23:48.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:48 smithi069 ceph-mon[94241]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:23:48.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:49.442 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 8443 2023-07-12T15:23:49.443 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.892+0000 7f7fc31a3700 1 Processor -- start 2023-07-12T15:23:49.443 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.892+0000 7f7fc31a3700 1 -- start start 2023-07-12T15:23:49.443 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.892+0000 7f7fc31a3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 0x7f7fbc104890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:49.443 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.892+0000 7f7fc31a3700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc104e60 con 0x7f7fbc104480 2023-07-12T15:23:49.443 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.892+0000 7f7fc0f3f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 0x7f7fbc104890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:49.443 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.893+0000 7f7fc0f3f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 0x7f7fbc104890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39838/0 (socket says 172.21.15.69:39838) 2023-07-12T15:23:49.443 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.893+0000 7f7fc0f3f700 1 -- 172.21.15.69:0/510897706 learned_addr learned my addr 172.21.15.69:0/510897706 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:49.444 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.893+0000 7f7fc0f3f700 1 -- 172.21.15.69:0/510897706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fbc104fa0 con 0x7f7fbc104480 2023-07-12T15:23:49.444 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.893+0000 7f7fc0f3f700 1 --2- 172.21.15.69:0/510897706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 0x7f7fbc104890 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000bb40 tx=0x7f7fb000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874a25fa0ce03e95 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:49.444 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.893+0000 7f7fbb7fe700 1 -- 172.21.15.69:0/510897706 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fb000dbe0 con 0x7f7fbc104480 2023-07-12T15:23:49.444 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.893+0000 7f7fbb7fe700 1 -- 172.21.15.69:0/510897706 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f7fb00044c0 con 0x7f7fbc104480 2023-07-12T15:23:49.444 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.893+0000 7f7fbb7fe700 1 -- 172.21.15.69:0/510897706 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fb0003d20 con 0x7f7fbc104480 2023-07-12T15:23:49.444 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.894+0000 7f7fc31a3700 1 -- 172.21.15.69:0/510897706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 msgr2=0x7f7fbc104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:49.444 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.894+0000 7f7fc31a3700 1 --2- 172.21.15.69:0/510897706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 0x7f7fbc104890 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000bb40 tx=0x7f7fb000be50 comp rx=0 tx=0).stop 2023-07-12T15:23:49.445 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.894+0000 7f7fc31a3700 1 -- 172.21.15.69:0/510897706 shutdown_connections 2023-07-12T15:23:49.445 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.894+0000 7f7fc31a3700 1 --2- 172.21.15.69:0/510897706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 0x7f7fbc104890 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:49.445 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.894+0000 7f7fc31a3700 1 -- 172.21.15.69:0/510897706 >> 172.21.15.69:0/510897706 conn(0x7f7fbc0fdca0 msgr2=0x7f7fbc1000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:49.445 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.894+0000 7f7fc31a3700 1 -- 172.21.15.69:0/510897706 shutdown_connections 2023-07-12T15:23:49.445 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.894+0000 7f7fc31a3700 1 -- 172.21.15.69:0/510897706 wait complete. 2023-07-12T15:23:49.445 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.894+0000 7f7fc31a3700 1 Processor -- start 2023-07-12T15:23:49.445 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.894+0000 7f7fc31a3700 1 -- start start 2023-07-12T15:23:49.446 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.894+0000 7f7fc31a3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 0x7f7fbc199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:49.446 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.894+0000 7f7fc31a3700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc19a3b0 con 0x7f7fbc104480 2023-07-12T15:23:49.446 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.895+0000 7f7fc0f3f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 0x7f7fbc199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:49.446 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.895+0000 7f7fc0f3f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 0x7f7fbc199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39840/0 (socket says 172.21.15.69:39840) 2023-07-12T15:23:49.446 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.895+0000 7f7fc0f3f700 1 -- 172.21.15.69:0/4188420890 learned_addr learned my addr 172.21.15.69:0/4188420890 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:49.446 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.895+0000 7f7fc0f3f700 1 -- 172.21.15.69:0/4188420890 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb000b7f0 con 0x7f7fbc104480 2023-07-12T15:23:49.446 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.895+0000 7f7fc0f3f700 1 --2- 172.21.15.69:0/4188420890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 0x7f7fbc199e70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00041a0 tx=0x7f7fb0004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:49.447 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.895+0000 7f7fb9ffb700 1 -- 172.21.15.69:0/4188420890 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fb00107b0 con 0x7f7fbc104480 2023-07-12T15:23:49.447 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.895+0000 7f7fb9ffb700 1 -- 172.21.15.69:0/4188420890 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f7fb0010da0 con 0x7f7fbc104480 2023-07-12T15:23:49.447 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.895+0000 7f7fc31a3700 1 -- 172.21.15.69:0/4188420890 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fbc19a5b0 con 0x7f7fbc104480 2023-07-12T15:23:49.447 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.895+0000 7f7fb9ffb700 1 -- 172.21.15.69:0/4188420890 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fb0021c50 con 0x7f7fbc104480 2023-07-12T15:23:49.447 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.895+0000 7f7fc31a3700 1 -- 172.21.15.69:0/4188420890 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fbc19a9f0 con 0x7f7fbc104480 2023-07-12T15:23:49.447 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.896+0000 7f7fb9ffb700 1 -- 172.21.15.69:0/4188420890 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 12) v1 ==== 45043+0+0 (secure 0 0 0) 0x7f7fb00193f0 con 0x7f7fbc104480 2023-07-12T15:23:49.447 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.896+0000 7f7fb9ffb700 1 --2- 172.21.15.69:0/4188420890 >> v2:172.21.15.69:6800/284433772 conn(0x7f7fac038420 0x7f7fac03a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:49.447 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.896+0000 7f7fb9ffb700 1 -- 172.21.15.69:0/4188420890 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f7fb004c9a0 con 0x7f7fbc104480 2023-07-12T15:23:49.448 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.896+0000 7f7fbbfff700 1 --2- 172.21.15.69:0/4188420890 >> v2:172.21.15.69:6800/284433772 conn(0x7f7fac038420 0x7f7fac03a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:49.448 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.896+0000 7f7fc31a3700 1 -- 172.21.15.69:0/4188420890 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fbc193c30 con 0x7f7fbc104480 2023-07-12T15:23:49.448 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.897+0000 7f7fbbfff700 1 --2- 172.21.15.69:0/4188420890 >> v2:172.21.15.69:6800/284433772 conn(0x7f7fac038420 0x7f7fac03a8d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8006fd0 tx=0x7f7fa8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:49.448 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:48.901+0000 7f7fb9ffb700 1 -- 172.21.15.69:0/4188420890 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7fb0010910 con 0x7f7fbc104480 2023-07-12T15:23:49.448 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.088+0000 7f7fc31a3700 1 -- 172.21.15.69:0/4188420890 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f7fbc02cf50 con 0x7f7fbc104480 2023-07-12T15:23:49.448 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.088+0000 7f7fb9ffb700 1 -- 172.21.15.69:0/4188420890 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f7fb000e360 con 0x7f7fbc104480 2023-07-12T15:23:49.449 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.090+0000 7f7fc31a3700 1 -- 172.21.15.69:0/4188420890 >> v2:172.21.15.69:6800/284433772 conn(0x7f7fac038420 msgr2=0x7f7fac03a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:49.449 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.090+0000 7f7fc31a3700 1 --2- 172.21.15.69:0/4188420890 >> v2:172.21.15.69:6800/284433772 conn(0x7f7fac038420 0x7f7fac03a8d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8006fd0 tx=0x7f7fa8006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:49.449 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.090+0000 7f7fc31a3700 1 -- 172.21.15.69:0/4188420890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 msgr2=0x7f7fbc199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:49.449 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.090+0000 7f7fc31a3700 1 --2- 172.21.15.69:0/4188420890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 0x7f7fbc199e70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00041a0 tx=0x7f7fb0004280 comp rx=0 tx=0).stop 2023-07-12T15:23:49.449 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.090+0000 7f7fc31a3700 1 -- 172.21.15.69:0/4188420890 shutdown_connections 2023-07-12T15:23:49.449 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.090+0000 7f7fc31a3700 1 --2- 172.21.15.69:0/4188420890 >> v2:172.21.15.69:6800/284433772 conn(0x7f7fac038420 0x7f7fac03a8d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:49.449 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.090+0000 7f7fc31a3700 1 --2- 172.21.15.69:0/4188420890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc104480 0x7f7fbc199e70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:49.449 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.090+0000 7f7fc31a3700 1 -- 172.21.15.69:0/4188420890 >> 172.21.15.69:0/4188420890 conn(0x7f7fbc0fdca0 msgr2=0x7f7fbc190d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:49.450 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.090+0000 7f7fc31a3700 1 -- 172.21.15.69:0/4188420890 shutdown_connections 2023-07-12T15:23:49.450 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.090+0000 7f7fc31a3700 1 -- 172.21.15.69:0/4188420890 wait complete. 2023-07-12T15:23:49.450 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-12T15:23:49.451 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout disabled 2023-07-12T15:23:49.457 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-12T15:23:49.457 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2023-07-12T15:23:49.458 INFO:teuthology.orchestra.run.smithi069.stdout:firewalld.service is not enabled 2023-07-12T15:23:49.458 INFO:teuthology.orchestra.run.smithi069.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-12T15:23:49.459 INFO:teuthology.orchestra.run.smithi069.stdout:Ceph Dashboard is now available at: 2023-07-12T15:23:49.459 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:23:49.459 INFO:teuthology.orchestra.run.smithi069.stdout: URL: https://smithi069.front.sepia.ceph.com:8443/ 2023-07-12T15:23:49.459 INFO:teuthology.orchestra.run.smithi069.stdout: User: admin 2023-07-12T15:23:49.459 INFO:teuthology.orchestra.run.smithi069.stdout: Password: e0pb4ybswb 2023-07-12T15:23:49.460 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:23:49.460 INFO:teuthology.orchestra.run.smithi069.stdout:Saving cluster configuration to /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config directory 2023-07-12T15:23:49.460 INFO:teuthology.orchestra.run.smithi069.stdout:Enabling autotune for osd_memory_target 2023-07-12T15:23:49.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4188420890' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-12T15:23:50.504 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.914+0000 7f54b7e4f700 1 Processor -- start 2023-07-12T15:23:50.504 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.914+0000 7f54b7e4f700 1 -- start start 2023-07-12T15:23:50.504 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.914+0000 7f54b7e4f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 0x7f54b0106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:50.504 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.914+0000 7f54b7e4f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b0077de0 con 0x7f54b0104af0 2023-07-12T15:23:50.505 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.915+0000 7f54b5beb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 0x7f54b0106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:50.505 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.915+0000 7f54b5beb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 0x7f54b0106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39844/0 (socket says 172.21.15.69:39844) 2023-07-12T15:23:50.505 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.915+0000 7f54b5beb700 1 -- 172.21.15.69:0/644253598 learned_addr learned my addr 172.21.15.69:0/644253598 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:50.506 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.915+0000 7f54b5beb700 1 -- 172.21.15.69:0/644253598 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b0077f20 con 0x7f54b0104af0 2023-07-12T15:23:50.507 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.915+0000 7f54b5beb700 1 --2- 172.21.15.69:0/644253598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 0x7f54b0106ed0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f54ac0098a0 tx=0x7f54ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9db760986d9c714 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:50.507 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.915+0000 7f54b4be9700 1 -- 172.21.15.69:0/644253598 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54ac004160 con 0x7f54b0104af0 2023-07-12T15:23:50.507 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.915+0000 7f54b4be9700 1 -- 172.21.15.69:0/644253598 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f54ac0042c0 con 0x7f54b0104af0 2023-07-12T15:23:50.507 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.915+0000 7f54b4be9700 1 -- 172.21.15.69:0/644253598 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54ac0118c0 con 0x7f54b0104af0 2023-07-12T15:23:50.507 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.916+0000 7f54b7e4f700 1 -- 172.21.15.69:0/644253598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 msgr2=0x7f54b0106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:50.507 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.916+0000 7f54b7e4f700 1 --2- 172.21.15.69:0/644253598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 0x7f54b0106ed0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f54ac0098a0 tx=0x7f54ac009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:50.508 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.916+0000 7f54b7e4f700 1 -- 172.21.15.69:0/644253598 shutdown_connections 2023-07-12T15:23:50.508 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.916+0000 7f54b7e4f700 1 --2- 172.21.15.69:0/644253598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 0x7f54b0106ed0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:50.508 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.916+0000 7f54b7e4f700 1 -- 172.21.15.69:0/644253598 >> 172.21.15.69:0/644253598 conn(0x7f54b00fe410 msgr2=0x7f54b0100880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:50.508 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.916+0000 7f54b7e4f700 1 -- 172.21.15.69:0/644253598 shutdown_connections 2023-07-12T15:23:50.508 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.916+0000 7f54b7e4f700 1 -- 172.21.15.69:0/644253598 wait complete. 2023-07-12T15:23:50.508 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.917+0000 7f54b7e4f700 1 Processor -- start 2023-07-12T15:23:50.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.917+0000 7f54b7e4f700 1 -- start start 2023-07-12T15:23:50.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.917+0000 7f54b7e4f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 0x7f54b0197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:50.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.917+0000 7f54b7e4f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b0198120 con 0x7f54b0104af0 2023-07-12T15:23:50.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.917+0000 7f54b5beb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 0x7f54b0197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:50.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.917+0000 7f54b5beb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 0x7f54b0197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39846/0 (socket says 172.21.15.69:39846) 2023-07-12T15:23:50.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.917+0000 7f54b5beb700 1 -- 172.21.15.69:0/3593462015 learned_addr learned my addr 172.21.15.69:0/3593462015 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:50.509 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.917+0000 7f54b5beb700 1 -- 172.21.15.69:0/3593462015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac009550 con 0x7f54b0104af0 2023-07-12T15:23:50.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.917+0000 7f54b5beb700 1 --2- 172.21.15.69:0/3593462015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 0x7f54b0197be0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f54ac0046c0 tx=0x7f54ac011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:50.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.917+0000 7f54a6ffd700 1 -- 172.21.15.69:0/3593462015 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54ac0036a0 con 0x7f54b0104af0 2023-07-12T15:23:50.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.918+0000 7f54a6ffd700 1 -- 172.21.15.69:0/3593462015 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f54ac003800 con 0x7f54b0104af0 2023-07-12T15:23:50.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.918+0000 7f54b7e4f700 1 -- 172.21.15.69:0/3593462015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b0198320 con 0x7f54b0104af0 2023-07-12T15:23:50.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.918+0000 7f54a6ffd700 1 -- 172.21.15.69:0/3593462015 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54ac01aea0 con 0x7f54b0104af0 2023-07-12T15:23:50.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.918+0000 7f54b7e4f700 1 -- 172.21.15.69:0/3593462015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b01987c0 con 0x7f54b0104af0 2023-07-12T15:23:50.510 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.918+0000 7f54a6ffd700 1 -- 172.21.15.69:0/3593462015 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7f54ac01a430 con 0x7f54b0104af0 2023-07-12T15:23:50.511 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.918+0000 7f54b7e4f700 1 -- 172.21.15.69:0/3593462015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b0108120 con 0x7f54b0104af0 2023-07-12T15:23:50.511 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.919+0000 7f54a6ffd700 1 --2- 172.21.15.69:0/3593462015 >> v2:172.21.15.69:6800/284433772 conn(0x7f549c038470 0x7f549c03a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:50.511 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.919+0000 7f54a6ffd700 1 -- 172.21.15.69:0/3593462015 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f54ac04c210 con 0x7f54b0104af0 2023-07-12T15:23:50.511 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.919+0000 7f54b53ea700 1 --2- 172.21.15.69:0/3593462015 >> v2:172.21.15.69:6800/284433772 conn(0x7f549c038470 0x7f549c03a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:50.511 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.919+0000 7f54b53ea700 1 --2- 172.21.15.69:0/3593462015 >> v2:172.21.15.69:6800/284433772 conn(0x7f549c038470 0x7f549c03a920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54a0006fd0 tx=0x7f54a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:50.511 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:49.923+0000 7f54a6ffd700 1 -- 172.21.15.69:0/3593462015 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f54ac023b80 con 0x7f54b0104af0 2023-07-12T15:23:50.511 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.107+0000 7f54b7e4f700 1 -- 172.21.15.69:0/3593462015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f54b0108310 con 0x7f54b0104af0 2023-07-12T15:23:50.512 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.108+0000 7f54a6ffd700 1 -- 172.21.15.69:0/3593462015 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f54ac023b80 con 0x7f54b0104af0 2023-07-12T15:23:50.512 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.110+0000 7f54b7e4f700 1 -- 172.21.15.69:0/3593462015 >> v2:172.21.15.69:6800/284433772 conn(0x7f549c038470 msgr2=0x7f549c03a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:50.512 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.110+0000 7f54b7e4f700 1 --2- 172.21.15.69:0/3593462015 >> v2:172.21.15.69:6800/284433772 conn(0x7f549c038470 0x7f549c03a920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54a0006fd0 tx=0x7f54a0006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:50.512 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.110+0000 7f54b7e4f700 1 -- 172.21.15.69:0/3593462015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 msgr2=0x7f54b0197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:50.512 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.110+0000 7f54b7e4f700 1 --2- 172.21.15.69:0/3593462015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 0x7f54b0197be0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f54ac0046c0 tx=0x7f54ac011920 comp rx=0 tx=0).stop 2023-07-12T15:23:50.512 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.110+0000 7f54b7e4f700 1 -- 172.21.15.69:0/3593462015 shutdown_connections 2023-07-12T15:23:50.512 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.110+0000 7f54b7e4f700 1 --2- 172.21.15.69:0/3593462015 >> v2:172.21.15.69:6800/284433772 conn(0x7f549c038470 0x7f549c03a920 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:50.513 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.110+0000 7f54b7e4f700 1 --2- 172.21.15.69:0/3593462015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b0104af0 0x7f54b0197be0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:50.513 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.110+0000 7f54b7e4f700 1 -- 172.21.15.69:0/3593462015 >> 172.21.15.69:0/3593462015 conn(0x7f54b00fe410 msgr2=0x7f54b00ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:50.513 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.110+0000 7f54b7e4f700 1 -- 172.21.15.69:0/3593462015 shutdown_connections 2023-07-12T15:23:50.513 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.110+0000 7f54b7e4f700 1 -- 172.21.15.69:0/3593462015 wait complete. 2023-07-12T15:23:50.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:50 smithi069 ceph-mon[94241]: mgrmap e13: y(active, since 6s) 2023-07-12T15:23:51.748 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.958+0000 7fcbbeae8700 1 Processor -- start 2023-07-12T15:23:51.749 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.958+0000 7fcbbeae8700 1 -- start start 2023-07-12T15:23:51.749 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.958+0000 7fcbbeae8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 0x7fcbb8106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:51.750 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.958+0000 7fcbbeae8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb8077d70 con 0x7fcbb8104b00 2023-07-12T15:23:51.750 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.958+0000 7fcbbc884700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 0x7fcbb8106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:51.750 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.959+0000 7fcbbc884700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 0x7fcbb8106ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39850/0 (socket says 172.21.15.69:39850) 2023-07-12T15:23:51.751 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.959+0000 7fcbbc884700 1 -- 172.21.15.69:0/176749162 learned_addr learned my addr 172.21.15.69:0/176749162 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:51.751 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.959+0000 7fcbbc884700 1 -- 172.21.15.69:0/176749162 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb8077eb0 con 0x7fcbb8104b00 2023-07-12T15:23:51.751 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.959+0000 7fcbbc884700 1 --2- 172.21.15.69:0/176749162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 0x7fcbb8106ee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcbac0098a0 tx=0x7fcbac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe36c9df8539837 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:51.751 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.959+0000 7fcbb77fe700 1 -- 172.21.15.69:0/176749162 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbac004160 con 0x7fcbb8104b00 2023-07-12T15:23:51.751 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.959+0000 7fcbb77fe700 1 -- 172.21.15.69:0/176749162 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fcbac0042c0 con 0x7fcbb8104b00 2023-07-12T15:23:51.752 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.959+0000 7fcbb77fe700 1 -- 172.21.15.69:0/176749162 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbac0118c0 con 0x7fcbb8104b00 2023-07-12T15:23:51.752 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.960+0000 7fcbbeae8700 1 -- 172.21.15.69:0/176749162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 msgr2=0x7fcbb8106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:51.753 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.960+0000 7fcbbeae8700 1 --2- 172.21.15.69:0/176749162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 0x7fcbb8106ee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcbac0098a0 tx=0x7fcbac009bb0 comp rx=0 tx=0).stop 2023-07-12T15:23:51.753 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.960+0000 7fcbbeae8700 1 -- 172.21.15.69:0/176749162 shutdown_connections 2023-07-12T15:23:51.753 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.960+0000 7fcbbeae8700 1 --2- 172.21.15.69:0/176749162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 0x7fcbb8106ee0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:51.753 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.960+0000 7fcbbeae8700 1 -- 172.21.15.69:0/176749162 >> 172.21.15.69:0/176749162 conn(0x7fcbb80fe540 msgr2=0x7fcbb8100950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:51.753 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.960+0000 7fcbbeae8700 1 -- 172.21.15.69:0/176749162 shutdown_connections 2023-07-12T15:23:51.753 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.960+0000 7fcbbeae8700 1 -- 172.21.15.69:0/176749162 wait complete. 2023-07-12T15:23:51.754 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.960+0000 7fcbbeae8700 1 Processor -- start 2023-07-12T15:23:51.754 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.960+0000 7fcbbeae8700 1 -- start start 2023-07-12T15:23:51.754 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.960+0000 7fcbbeae8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 0x7fcbb8104460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:51.754 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.960+0000 7fcbbeae8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb81049a0 con 0x7fcbb8104b00 2023-07-12T15:23:51.754 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.961+0000 7fcbbc884700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 0x7fcbb8104460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:51.754 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.961+0000 7fcbbc884700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 0x7fcbb8104460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39852/0 (socket says 172.21.15.69:39852) 2023-07-12T15:23:51.754 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.961+0000 7fcbbc884700 1 -- 172.21.15.69:0/633324361 learned_addr learned my addr 172.21.15.69:0/633324361 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:51.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.961+0000 7fcbbc884700 1 -- 172.21.15.69:0/633324361 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac009550 con 0x7fcbb8104b00 2023-07-12T15:23:51.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.961+0000 7fcbbc884700 1 --2- 172.21.15.69:0/633324361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 0x7fcbb8104460 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcbac0046c0 tx=0x7fcbac011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:51.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.961+0000 7fcbb5ffb700 1 -- 172.21.15.69:0/633324361 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbac0036a0 con 0x7fcbb8104b00 2023-07-12T15:23:51.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.961+0000 7fcbb5ffb700 1 -- 172.21.15.69:0/633324361 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fcbac003800 con 0x7fcbb8104b00 2023-07-12T15:23:51.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.961+0000 7fcbb5ffb700 1 -- 172.21.15.69:0/633324361 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbac024e70 con 0x7fcbb8104b00 2023-07-12T15:23:51.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.961+0000 7fcbbeae8700 1 -- 172.21.15.69:0/633324361 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb8102ab0 con 0x7fcbb8104b00 2023-07-12T15:23:51.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.961+0000 7fcbbeae8700 1 -- 172.21.15.69:0/633324361 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb8102f50 con 0x7fcbb8104b00 2023-07-12T15:23:51.755 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.962+0000 7fcbb5ffb700 1 -- 172.21.15.69:0/633324361 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7fcbac01a480 con 0x7fcbb8104b00 2023-07-12T15:23:51.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.962+0000 7fcbb5ffb700 1 --2- 172.21.15.69:0/633324361 >> v2:172.21.15.69:6800/284433772 conn(0x7fcba8038410 0x7fcba803a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:51.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.962+0000 7fcbbeae8700 1 -- 172.21.15.69:0/633324361 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb8062130 con 0x7fcbb8104b00 2023-07-12T15:23:51.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.962+0000 7fcbb5ffb700 1 -- 172.21.15.69:0/633324361 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fcbac04c1c0 con 0x7fcbb8104b00 2023-07-12T15:23:51.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.962+0000 7fcbb7fff700 1 --2- 172.21.15.69:0/633324361 >> v2:172.21.15.69:6800/284433772 conn(0x7fcba8038410 0x7fcba803a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:51.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.963+0000 7fcbb7fff700 1 --2- 172.21.15.69:0/633324361 >> v2:172.21.15.69:6800/284433772 conn(0x7fcba8038410 0x7fcba803a8c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcba4006fd0 tx=0x7fcba4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:51.756 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:50.967+0000 7fcbb5ffb700 1 -- 172.21.15.69:0/633324361 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcbac0189a0 con 0x7fcbb8104b00 2023-07-12T15:23:51.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:51.256+0000 7fcbbeae8700 1 -- 172.21.15.69:0/633324361 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fcbb8103190 con 0x7fcbb8104b00 2023-07-12T15:23:51.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:51.258+0000 7fcbb5ffb700 1 -- 172.21.15.69:0/633324361 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fcbac0187c0 con 0x7fcbb8104b00 2023-07-12T15:23:51.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:51.261+0000 7fcbbeae8700 1 -- 172.21.15.69:0/633324361 >> v2:172.21.15.69:6800/284433772 conn(0x7fcba8038410 msgr2=0x7fcba803a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:51.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:51.261+0000 7fcbbeae8700 1 --2- 172.21.15.69:0/633324361 >> v2:172.21.15.69:6800/284433772 conn(0x7fcba8038410 0x7fcba803a8c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcba4006fd0 tx=0x7fcba4006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:51.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:51.261+0000 7fcbbeae8700 1 -- 172.21.15.69:0/633324361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 msgr2=0x7fcbb8104460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:51.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:51.261+0000 7fcbbeae8700 1 --2- 172.21.15.69:0/633324361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 0x7fcbb8104460 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcbac0046c0 tx=0x7fcbac011920 comp rx=0 tx=0).stop 2023-07-12T15:23:51.757 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:51.261+0000 7fcbbeae8700 1 -- 172.21.15.69:0/633324361 shutdown_connections 2023-07-12T15:23:51.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:51.261+0000 7fcbbeae8700 1 --2- 172.21.15.69:0/633324361 >> v2:172.21.15.69:6800/284433772 conn(0x7fcba8038410 0x7fcba803a8c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:51.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:51.261+0000 7fcbbeae8700 1 --2- 172.21.15.69:0/633324361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbb8104b00 0x7fcbb8104460 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:51.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:51.261+0000 7fcbbeae8700 1 -- 172.21.15.69:0/633324361 >> 172.21.15.69:0/633324361 conn(0x7fcbb80fe540 msgr2=0x7fcbb80ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:51.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:51.261+0000 7fcbbeae8700 1 -- 172.21.15.69:0/633324361 shutdown_connections 2023-07-12T15:23:51.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2023-07-12T15:23:51.261+0000 7fcbbeae8700 1 -- 172.21.15.69:0/633324361 wait complete. 2023-07-12T15:23:51.758 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-12T15:23:51.758 INFO:teuthology.orchestra.run.smithi069.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-12T15:23:51.759 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:23:51.759 INFO:teuthology.orchestra.run.smithi069.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-12T15:23:51.759 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:23:51.759 INFO:teuthology.orchestra.run.smithi069.stdout:Or, if you are only running a single cluster on this host: 2023-07-12T15:23:51.759 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:23:51.759 INFO:teuthology.orchestra.run.smithi069.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-12T15:23:51.759 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:23:51.760 INFO:teuthology.orchestra.run.smithi069.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-12T15:23:51.760 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:23:51.760 INFO:teuthology.orchestra.run.smithi069.stdout: ceph telemetry on 2023-07-12T15:23:51.760 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:23:51.760 INFO:teuthology.orchestra.run.smithi069.stdout:For more information see: 2023-07-12T15:23:51.760 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:23:51.760 INFO:teuthology.orchestra.run.smithi069.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-12T15:23:51.761 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:23:51.761 INFO:teuthology.orchestra.run.smithi069.stdout:Bootstrap complete. 2023-07-12T15:23:51.779 INFO:tasks.cephadm:Fetching config... 2023-07-12T15:23:51.780 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:23:51.780 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-12T15:23:51.798 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-12T15:23:51.799 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:23:51.799 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-12T15:23:51.856 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-12T15:23:51.857 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:23:51.858 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-12T15:23:51.924 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-12T15:23:51.925 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:23:51.925 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-12T15:23:51.981 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-12T15:23:51.981 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx5AZBZo8AuiArArcXOFtG4+RnBzbgzZdifySyikikFatg/V+eSMp0CnjS7FiTJ78OlRtDWHqyQEuwhAqNfn/q9pznEvgP2XXiS6aaOuSuXmj3mvt0v2ROtVcs/z0dFu5QPD0wxPcJ9ZmFDWCipLqT+cd7muGV3/VDx8VYtrRFoiu2pyFlnVkV15HkkUN9yWT4QjusKWIUjEmVdHzIz4yzsSNR2XOCu8+WRP+pCTs+Z+10DtXVzBb5qHYCYHEAlPkfg2P88BxbkPC9JBAj5A7iFjjYIe5qLA7a2z2xX/gCZxZgvv6S6FwssNVXiZfkc+b2kWGXDY9A4eRU8uKfAoc4QYDwoJStH069gnbuJlYuxKNugm5URUA8WnUbYVcNhTXaN2jdulG9Jvhm3vQMI5o4BeTcey8sVaWgasEHPK/UfhqrfEriiABaBn2IIKItsZ8Tycm2cEQFfg2Ik6w4HHv9uScPBkoYGgeQybvHejM68B2Uqqyq3SQ602I8n8el0UM= ceph-dc75a490-20c7-11ee-9b31-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-12T15:23:52.063 INFO:teuthology.orchestra.run.smithi069.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx5AZBZo8AuiArArcXOFtG4+RnBzbgzZdifySyikikFatg/V+eSMp0CnjS7FiTJ78OlRtDWHqyQEuwhAqNfn/q9pznEvgP2XXiS6aaOuSuXmj3mvt0v2ROtVcs/z0dFu5QPD0wxPcJ9ZmFDWCipLqT+cd7muGV3/VDx8VYtrRFoiu2pyFlnVkV15HkkUN9yWT4QjusKWIUjEmVdHzIz4yzsSNR2XOCu8+WRP+pCTs+Z+10DtXVzBb5qHYCYHEAlPkfg2P88BxbkPC9JBAj5A7iFjjYIe5qLA7a2z2xX/gCZxZgvv6S6FwssNVXiZfkc+b2kWGXDY9A4eRU8uKfAoc4QYDwoJStH069gnbuJlYuxKNugm5URUA8WnUbYVcNhTXaN2jdulG9Jvhm3vQMI5o4BeTcey8sVaWgasEHPK/UfhqrfEriiABaBn2IIKItsZ8Tycm2cEQFfg2Ik6w4HHv9uScPBkoYGgeQybvHejM68B2Uqqyq3SQ602I8n8el0UM= ceph-dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:23:52.076 DEBUG:teuthology.orchestra.run.smithi145:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx5AZBZo8AuiArArcXOFtG4+RnBzbgzZdifySyikikFatg/V+eSMp0CnjS7FiTJ78OlRtDWHqyQEuwhAqNfn/q9pznEvgP2XXiS6aaOuSuXmj3mvt0v2ROtVcs/z0dFu5QPD0wxPcJ9ZmFDWCipLqT+cd7muGV3/VDx8VYtrRFoiu2pyFlnVkV15HkkUN9yWT4QjusKWIUjEmVdHzIz4yzsSNR2XOCu8+WRP+pCTs+Z+10DtXVzBb5qHYCYHEAlPkfg2P88BxbkPC9JBAj5A7iFjjYIe5qLA7a2z2xX/gCZxZgvv6S6FwssNVXiZfkc+b2kWGXDY9A4eRU8uKfAoc4QYDwoJStH069gnbuJlYuxKNugm5URUA8WnUbYVcNhTXaN2jdulG9Jvhm3vQMI5o4BeTcey8sVaWgasEHPK/UfhqrfEriiABaBn2IIKItsZ8Tycm2cEQFfg2Ik6w4HHv9uScPBkoYGgeQybvHejM68B2Uqqyq3SQ602I8n8el0UM= ceph-dc75a490-20c7-11ee-9b31-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-12T15:23:52.123 INFO:teuthology.orchestra.run.smithi145.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx5AZBZo8AuiArArcXOFtG4+RnBzbgzZdifySyikikFatg/V+eSMp0CnjS7FiTJ78OlRtDWHqyQEuwhAqNfn/q9pznEvgP2XXiS6aaOuSuXmj3mvt0v2ROtVcs/z0dFu5QPD0wxPcJ9ZmFDWCipLqT+cd7muGV3/VDx8VYtrRFoiu2pyFlnVkV15HkkUN9yWT4QjusKWIUjEmVdHzIz4yzsSNR2XOCu8+WRP+pCTs+Z+10DtXVzBb5qHYCYHEAlPkfg2P88BxbkPC9JBAj5A7iFjjYIe5qLA7a2z2xX/gCZxZgvv6S6FwssNVXiZfkc+b2kWGXDY9A4eRU8uKfAoc4QYDwoJStH069gnbuJlYuxKNugm5URUA8WnUbYVcNhTXaN2jdulG9Jvhm3vQMI5o4BeTcey8sVaWgasEHPK/UfhqrfEriiABaBn2IIKItsZ8Tycm2cEQFfg2Ik6w4HHv9uScPBkoYGgeQybvHejM68B2Uqqyq3SQ602I8n8el0UM= ceph-dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:23:52.135 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-12T15:23:52.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/633324361' entity='client.admin' 2023-07-12T15:23:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:23:54.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:54.141 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:54.141 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi069", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:23:54.141 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:54.141 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:23:54.141 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:54.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.377+0000 7fba53834700 1 -- 172.21.15.69:0/2803037117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c0ffbd0 msgr2=0x7fba4c0fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:54.379 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.377+0000 7fba53834700 1 --2- 172.21.15.69:0/2803037117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c0ffbd0 0x7fba4c0fffe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fba48009910 tx=0x7fba48009c20 comp rx=0 tx=0).stop 2023-07-12T15:23:54.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.377+0000 7fba53834700 1 -- 172.21.15.69:0/2803037117 shutdown_connections 2023-07-12T15:23:54.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.377+0000 7fba53834700 1 --2- 172.21.15.69:0/2803037117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c0ffbd0 0x7fba4c0fffe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:54.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.377+0000 7fba53834700 1 -- 172.21.15.69:0/2803037117 >> 172.21.15.69:0/2803037117 conn(0x7fba4c0fb220 msgr2=0x7fba4c0fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:54.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.377+0000 7fba53834700 1 -- 172.21.15.69:0/2803037117 shutdown_connections 2023-07-12T15:23:54.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.377+0000 7fba53834700 1 -- 172.21.15.69:0/2803037117 wait complete. 2023-07-12T15:23:54.380 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.378+0000 7fba53834700 1 Processor -- start 2023-07-12T15:23:54.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.378+0000 7fba53834700 1 -- start start 2023-07-12T15:23:54.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.378+0000 7fba53834700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c0ffbd0 0x7fba4c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:54.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.378+0000 7fba53834700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c074de0 con 0x7fba4c0ffbd0 2023-07-12T15:23:54.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.378+0000 7fba515d0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c0ffbd0 0x7fba4c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:54.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.378+0000 7fba515d0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c0ffbd0 0x7fba4c0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39858/0 (socket says 172.21.15.69:39858) 2023-07-12T15:23:54.381 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.378+0000 7fba515d0700 1 -- 172.21.15.69:0/1773256479 learned_addr learned my addr 172.21.15.69:0/1773256479 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:54.382 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.379+0000 7fba515d0700 1 -- 172.21.15.69:0/1773256479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba480095f0 con 0x7fba4c0ffbd0 2023-07-12T15:23:54.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.379+0000 7fba515d0700 1 --2- 172.21.15.69:0/1773256479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c0ffbd0 0x7fba4c0748a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fba48005f50 tx=0x7fba48004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:54.383 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.379+0000 7fba427fc700 1 -- 172.21.15.69:0/1773256479 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba4800ba90 con 0x7fba4c0ffbd0 2023-07-12T15:23:54.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.379+0000 7fba427fc700 1 -- 172.21.15.69:0/1773256479 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fba4800bbf0 con 0x7fba4c0ffbd0 2023-07-12T15:23:54.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.379+0000 7fba53834700 1 -- 172.21.15.69:0/1773256479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4c072ef0 con 0x7fba4c0ffbd0 2023-07-12T15:23:54.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.379+0000 7fba427fc700 1 -- 172.21.15.69:0/1773256479 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba48017440 con 0x7fba4c0ffbd0 2023-07-12T15:23:54.385 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.379+0000 7fba53834700 1 -- 172.21.15.69:0/1773256479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba4c073390 con 0x7fba4c0ffbd0 2023-07-12T15:23:54.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.380+0000 7fba427fc700 1 -- 172.21.15.69:0/1773256479 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7fba48017700 con 0x7fba4c0ffbd0 2023-07-12T15:23:54.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.380+0000 7fba427fc700 1 --2- 172.21.15.69:0/1773256479 >> v2:172.21.15.69:6800/284433772 conn(0x7fba38038420 0x7fba3803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:54.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.380+0000 7fba53834700 1 -- 172.21.15.69:0/1773256479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4c073da0 con 0x7fba4c0ffbd0 2023-07-12T15:23:54.386 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.380+0000 7fba427fc700 1 -- 172.21.15.69:0/1773256479 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fba4804c770 con 0x7fba4c0ffbd0 2023-07-12T15:23:54.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.380+0000 7fba50dcf700 1 --2- 172.21.15.69:0/1773256479 >> v2:172.21.15.69:6800/284433772 conn(0x7fba38038420 0x7fba3803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:54.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.380+0000 7fba50dcf700 1 --2- 172.21.15.69:0/1773256479 >> v2:172.21.15.69:6800/284433772 conn(0x7fba38038420 0x7fba3803a8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fba3c006fd0 tx=0x7fba3c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:54.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.385+0000 7fba427fc700 1 -- 172.21.15.69:0/1773256479 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fba4802c440 con 0x7fba4c0ffbd0 2023-07-12T15:23:54.573 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.572+0000 7fba53834700 1 -- 172.21.15.69:0/1773256479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fba4c062130 con 0x7fba4c0ffbd0 2023-07-12T15:23:54.577 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.576+0000 7fba427fc700 1 -- 172.21.15.69:0/1773256479 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fba48027020 con 0x7fba4c0ffbd0 2023-07-12T15:23:54.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.578+0000 7fba53834700 1 -- 172.21.15.69:0/1773256479 >> v2:172.21.15.69:6800/284433772 conn(0x7fba38038420 msgr2=0x7fba3803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:54.580 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.578+0000 7fba53834700 1 --2- 172.21.15.69:0/1773256479 >> v2:172.21.15.69:6800/284433772 conn(0x7fba38038420 0x7fba3803a8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fba3c006fd0 tx=0x7fba3c006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:54.581 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.578+0000 7fba53834700 1 -- 172.21.15.69:0/1773256479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c0ffbd0 msgr2=0x7fba4c0748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:54.581 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.578+0000 7fba53834700 1 --2- 172.21.15.69:0/1773256479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c0ffbd0 0x7fba4c0748a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fba48005f50 tx=0x7fba48004f20 comp rx=0 tx=0).stop 2023-07-12T15:23:54.581 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.578+0000 7fba53834700 1 -- 172.21.15.69:0/1773256479 shutdown_connections 2023-07-12T15:23:54.581 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.578+0000 7fba53834700 1 --2- 172.21.15.69:0/1773256479 >> v2:172.21.15.69:6800/284433772 conn(0x7fba38038420 0x7fba3803a8d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:54.581 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.578+0000 7fba53834700 1 --2- 172.21.15.69:0/1773256479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba4c0ffbd0 0x7fba4c0748a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:54.581 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.578+0000 7fba53834700 1 -- 172.21.15.69:0/1773256479 >> 172.21.15.69:0/1773256479 conn(0x7fba4c0fb220 msgr2=0x7fba4c0fbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:54.581 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.578+0000 7fba53834700 1 -- 172.21.15.69:0/1773256479 shutdown_connections 2023-07-12T15:23:54.582 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:54.578+0000 7fba53834700 1 -- 172.21.15.69:0/1773256479 wait complete. 2023-07-12T15:23:54.981 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-12T15:23:54.982 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-12T15:23:55.531 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:23:55.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1773256479' entity='client.admin' 2023-07-12T15:23:55.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:55 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:23:55.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:55 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:56.689 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.687+0000 7fd2901cf700 1 -- 172.21.15.69:0/4078172196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2880fead0 msgr2=0x7fd2880feee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:56.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.687+0000 7fd2901cf700 1 --2- 172.21.15.69:0/4078172196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2880fead0 0x7fd2880feee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009910 tx=0x7fd27c009c20 comp rx=0 tx=0).stop 2023-07-12T15:23:56.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.687+0000 7fd2901cf700 1 -- 172.21.15.69:0/4078172196 shutdown_connections 2023-07-12T15:23:56.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.687+0000 7fd2901cf700 1 --2- 172.21.15.69:0/4078172196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2880fead0 0x7fd2880feee0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:56.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.687+0000 7fd2901cf700 1 -- 172.21.15.69:0/4078172196 >> 172.21.15.69:0/4078172196 conn(0x7fd2880fa060 msgr2=0x7fd2880fc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:56.690 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.687+0000 7fd2901cf700 1 -- 172.21.15.69:0/4078172196 shutdown_connections 2023-07-12T15:23:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.687+0000 7fd2901cf700 1 -- 172.21.15.69:0/4078172196 wait complete. 2023-07-12T15:23:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.688+0000 7fd2901cf700 1 Processor -- start 2023-07-12T15:23:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.688+0000 7fd2901cf700 1 -- start start 2023-07-12T15:23:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.688+0000 7fd2901cf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2880fead0 0x7fd288193330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.688+0000 7fd2901cf700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd288193870 con 0x7fd2880fead0 2023-07-12T15:23:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.688+0000 7fd28df6b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2880fead0 0x7fd288193330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.688+0000 7fd28df6b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2880fead0 0x7fd288193330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39864/0 (socket says 172.21.15.69:39864) 2023-07-12T15:23:56.691 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.688+0000 7fd28df6b700 1 -- 172.21.15.69:0/221158418 learned_addr learned my addr 172.21.15.69:0/221158418 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:56.692 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.689+0000 7fd28df6b700 1 -- 172.21.15.69:0/221158418 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c0095f0 con 0x7fd2880fead0 2023-07-12T15:23:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.689+0000 7fd28df6b700 1 --2- 172.21.15.69:0/221158418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2880fead0 0x7fd288193330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd27c005740 tx=0x7fd27c004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.689+0000 7fd27affd700 1 -- 172.21.15.69:0/221158418 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd27c00b960 con 0x7fd2880fead0 2023-07-12T15:23:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.689+0000 7fd27affd700 1 -- 172.21.15.69:0/221158418 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fd27c00f460 con 0x7fd2880fead0 2023-07-12T15:23:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.689+0000 7fd2901cf700 1 -- 172.21.15.69:0/221158418 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd288193a70 con 0x7fd2880fead0 2023-07-12T15:23:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.689+0000 7fd27affd700 1 -- 172.21.15.69:0/221158418 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd27c017440 con 0x7fd2880fead0 2023-07-12T15:23:56.693 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.689+0000 7fd2901cf700 1 -- 172.21.15.69:0/221158418 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd288193f10 con 0x7fd2880fead0 2023-07-12T15:23:56.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.690+0000 7fd27affd700 1 -- 172.21.15.69:0/221158418 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7fd27c017630 con 0x7fd2880fead0 2023-07-12T15:23:56.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.690+0000 7fd2901cf700 1 -- 172.21.15.69:0/221158418 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28818cd00 con 0x7fd2880fead0 2023-07-12T15:23:56.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.690+0000 7fd27affd700 1 --2- 172.21.15.69:0/221158418 >> v2:172.21.15.69:6800/284433772 conn(0x7fd274038470 0x7fd27403a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:56.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.690+0000 7fd27affd700 1 -- 172.21.15.69:0/221158418 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fd27c04c480 con 0x7fd2880fead0 2023-07-12T15:23:56.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.690+0000 7fd28d76a700 1 --2- 172.21.15.69:0/221158418 >> v2:172.21.15.69:6800/284433772 conn(0x7fd274038470 0x7fd27403a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:56.694 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.690+0000 7fd28d76a700 1 --2- 172.21.15.69:0/221158418 >> v2:172.21.15.69:6800/284433772 conn(0x7fd274038470 0x7fd27403a920 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd284006fd0 tx=0x7fd284006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:56.696 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.695+0000 7fd27affd700 1 -- 172.21.15.69:0/221158418 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd27c015350 con 0x7fd2880fead0 2023-07-12T15:23:56.885 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.884+0000 7fd2901cf700 1 -- 172.21.15.69:0/221158418 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fd288060f40 con 0x7fd274038470 2023-07-12T15:23:56.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.887+0000 7fd27affd700 1 -- 172.21.15.69:0/221158418 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd288060f40 con 0x7fd274038470 2023-07-12T15:23:56.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.889+0000 7fd2901cf700 1 -- 172.21.15.69:0/221158418 >> v2:172.21.15.69:6800/284433772 conn(0x7fd274038470 msgr2=0x7fd27403a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:56.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.889+0000 7fd2901cf700 1 --2- 172.21.15.69:0/221158418 >> v2:172.21.15.69:6800/284433772 conn(0x7fd274038470 0x7fd27403a920 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd284006fd0 tx=0x7fd284006e40 comp rx=0 tx=0).stop 2023-07-12T15:23:56.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.889+0000 7fd2901cf700 1 -- 172.21.15.69:0/221158418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2880fead0 msgr2=0x7fd288193330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:56.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.889+0000 7fd2901cf700 1 --2- 172.21.15.69:0/221158418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2880fead0 0x7fd288193330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd27c005740 tx=0x7fd27c004f20 comp rx=0 tx=0).stop 2023-07-12T15:23:56.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.889+0000 7fd2901cf700 1 -- 172.21.15.69:0/221158418 shutdown_connections 2023-07-12T15:23:56.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.889+0000 7fd2901cf700 1 --2- 172.21.15.69:0/221158418 >> v2:172.21.15.69:6800/284433772 conn(0x7fd274038470 0x7fd27403a920 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:56.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.889+0000 7fd2901cf700 1 --2- 172.21.15.69:0/221158418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2880fead0 0x7fd288193330 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:56.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.889+0000 7fd2901cf700 1 -- 172.21.15.69:0/221158418 >> 172.21.15.69:0/221158418 conn(0x7fd2880fa060 msgr2=0x7fd2880fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:56.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.890+0000 7fd2901cf700 1 -- 172.21.15.69:0/221158418 shutdown_connections 2023-07-12T15:23:56.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:56.890+0000 7fd2901cf700 1 -- 172.21.15.69:0/221158418 wait complete. 2023-07-12T15:23:57.376 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi145 2023-07-12T15:23:57.376 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-07-12T15:23:57.377 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/etc/ceph/ceph.conf 2023-07-12T15:23:57.398 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-07-12T15:23:57.398 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-12T15:23:57.456 INFO:tasks.cephadm:Adding host smithi145 to orchestrator... 2023-07-12T15:23:57.457 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch host add smithi145 2023-07-12T15:23:58.027 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:23:58.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:57 smithi069 ceph-mon[94241]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:23:58.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:57 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:58.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:57 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:23:58.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:57 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:23:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:57 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:23:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:57 smithi069 ceph-mon[94241]: Updating smithi069:/etc/ceph/ceph.conf 2023-07-12T15:23:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:57 smithi069 ceph-mon[94241]: Updating smithi069:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:23:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:57 smithi069 ceph-mon[94241]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2023-07-12T15:23:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:57 smithi069 ceph-mon[94241]: Updating smithi069:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.client.admin.keyring 2023-07-12T15:23:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:57 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:57 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:57 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:23:59.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.180+0000 7ffb26f5a700 1 -- 172.21.15.69:0/766147680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb200ffb50 msgr2=0x7ffb200fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:23:59.183 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.180+0000 7ffb26f5a700 1 --2- 172.21.15.69:0/766147680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb200ffb50 0x7ffb200fff20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009910 tx=0x7ffb14009c20 comp rx=0 tx=0).stop 2023-07-12T15:23:59.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.180+0000 7ffb26f5a700 1 -- 172.21.15.69:0/766147680 shutdown_connections 2023-07-12T15:23:59.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.180+0000 7ffb26f5a700 1 --2- 172.21.15.69:0/766147680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb200ffb50 0x7ffb200fff20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:23:59.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.180+0000 7ffb26f5a700 1 -- 172.21.15.69:0/766147680 >> 172.21.15.69:0/766147680 conn(0x7ffb20075500 msgr2=0x7ffb20075900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:23:59.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.180+0000 7ffb26f5a700 1 -- 172.21.15.69:0/766147680 shutdown_connections 2023-07-12T15:23:59.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.180+0000 7ffb26f5a700 1 -- 172.21.15.69:0/766147680 wait complete. 2023-07-12T15:23:59.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.180+0000 7ffb26f5a700 1 Processor -- start 2023-07-12T15:23:59.184 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.180+0000 7ffb26f5a700 1 -- start start 2023-07-12T15:23:59.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.181+0000 7ffb26f5a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb200ffb50 0x7ffb2019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:59.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.181+0000 7ffb26f5a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2019ab00 con 0x7ffb200ffb50 2023-07-12T15:23:59.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.181+0000 7ffb24cf6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb200ffb50 0x7ffb2019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:59.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.181+0000 7ffb24cf6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb200ffb50 0x7ffb2019a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39870/0 (socket says 172.21.15.69:39870) 2023-07-12T15:23:59.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.181+0000 7ffb24cf6700 1 -- 172.21.15.69:0/561131860 learned_addr learned my addr 172.21.15.69:0/561131860 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:23:59.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.181+0000 7ffb24cf6700 1 -- 172.21.15.69:0/561131860 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb140095f0 con 0x7ffb200ffb50 2023-07-12T15:23:59.185 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.181+0000 7ffb24cf6700 1 --2- 172.21.15.69:0/561131860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb200ffb50 0x7ffb2019a5c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffb14004f40 tx=0x7ffb14005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:59.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.181+0000 7ffb1dffb700 1 -- 172.21.15.69:0/561131860 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffb1401d070 con 0x7ffb200ffb50 2023-07-12T15:23:59.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.181+0000 7ffb1dffb700 1 -- 172.21.15.69:0/561131860 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7ffb1400bdf0 con 0x7ffb200ffb50 2023-07-12T15:23:59.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.181+0000 7ffb26f5a700 1 -- 172.21.15.69:0/561131860 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb2019ad00 con 0x7ffb200ffb50 2023-07-12T15:23:59.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.181+0000 7ffb1dffb700 1 -- 172.21.15.69:0/561131860 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffb14022c00 con 0x7ffb200ffb50 2023-07-12T15:23:59.186 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.181+0000 7ffb26f5a700 1 -- 172.21.15.69:0/561131860 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb2019b060 con 0x7ffb200ffb50 2023-07-12T15:23:59.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.182+0000 7ffb1dffb700 1 -- 172.21.15.69:0/561131860 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7ffb1400f460 con 0x7ffb200ffb50 2023-07-12T15:23:59.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.182+0000 7ffb1dffb700 1 --2- 172.21.15.69:0/561131860 >> v2:172.21.15.69:6800/284433772 conn(0x7ffb10038420 0x7ffb1003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:23:59.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.182+0000 7ffb26f5a700 1 -- 172.21.15.69:0/561131860 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb20062130 con 0x7ffb200ffb50 2023-07-12T15:23:59.187 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.182+0000 7ffb1dffb700 1 -- 172.21.15.69:0/561131860 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7ffb1404c3d0 con 0x7ffb200ffb50 2023-07-12T15:23:59.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.182+0000 7ffb1ffff700 1 --2- 172.21.15.69:0/561131860 >> v2:172.21.15.69:6800/284433772 conn(0x7ffb10038420 0x7ffb1003a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:23:59.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.183+0000 7ffb1ffff700 1 --2- 172.21.15.69:0/561131860 >> v2:172.21.15.69:6800/284433772 conn(0x7ffb10038420 0x7ffb1003a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c006fd0 tx=0x7ffb0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:23:59.188 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.187+0000 7ffb1dffb700 1 -- 172.21.15.69:0/561131860 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ffb14020730 con 0x7ffb200ffb50 2023-07-12T15:23:59.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:23:59.377+0000 7ffb26f5a700 1 -- 172.21.15.69:0/561131860 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi145", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb20104030 con 0x7ffb10038420 2023-07-12T15:24:00.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:23:59 smithi069 ceph-mon[94241]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi145", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:24:01.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:00 smithi069 ceph-mon[94241]: Deploying cephadm binary to smithi145 2023-07-12T15:24:01.650 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:01.649+0000 7ffb1dffb700 1 -- 172.21.15.69:0/561131860 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ffb20104030 con 0x7ffb10038420 2023-07-12T15:24:01.651 INFO:teuthology.orchestra.run.smithi069.stdout:Added host 'smithi145' with addr '172.21.15.145' 2023-07-12T15:24:01.653 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:01.651+0000 7ffb26f5a700 1 -- 172.21.15.69:0/561131860 >> v2:172.21.15.69:6800/284433772 conn(0x7ffb10038420 msgr2=0x7ffb1003a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:01.653 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:01.651+0000 7ffb26f5a700 1 --2- 172.21.15.69:0/561131860 >> v2:172.21.15.69:6800/284433772 conn(0x7ffb10038420 0x7ffb1003a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c006fd0 tx=0x7ffb0c006e40 comp rx=0 tx=0).stop 2023-07-12T15:24:01.653 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:01.651+0000 7ffb26f5a700 1 -- 172.21.15.69:0/561131860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb200ffb50 msgr2=0x7ffb2019a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:01.654 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:01.651+0000 7ffb26f5a700 1 --2- 172.21.15.69:0/561131860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb200ffb50 0x7ffb2019a5c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffb14004f40 tx=0x7ffb14005020 comp rx=0 tx=0).stop 2023-07-12T15:24:01.654 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:01.651+0000 7ffb26f5a700 1 -- 172.21.15.69:0/561131860 shutdown_connections 2023-07-12T15:24:01.654 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:01.651+0000 7ffb26f5a700 1 --2- 172.21.15.69:0/561131860 >> v2:172.21.15.69:6800/284433772 conn(0x7ffb10038420 0x7ffb1003a8d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:01.654 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:01.651+0000 7ffb26f5a700 1 --2- 172.21.15.69:0/561131860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb200ffb50 0x7ffb2019a5c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:01.655 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:01.651+0000 7ffb26f5a700 1 -- 172.21.15.69:0/561131860 >> 172.21.15.69:0/561131860 conn(0x7ffb20075500 msgr2=0x7ffb20102910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:01.655 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:01.651+0000 7ffb26f5a700 1 -- 172.21.15.69:0/561131860 shutdown_connections 2023-07-12T15:24:01.655 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:01.651+0000 7ffb26f5a700 1 -- 172.21.15.69:0/561131860 wait complete. 2023-07-12T15:24:02.081 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch host ls --format=json 2023-07-12T15:24:02.632 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:24:02.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:02 smithi069 ceph-mon[94241]: Added host smithi145 2023-07-12T15:24:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:03.842 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.839+0000 7fd073603700 1 -- 172.21.15.69:0/3419490590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd06c104ae0 msgr2=0x7fd06c104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:03.842 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.839+0000 7fd073603700 1 --2- 172.21.15.69:0/3419490590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd06c104ae0 0x7fd06c104eb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd068009910 tx=0x7fd068009c20 comp rx=0 tx=0).stop 2023-07-12T15:24:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.839+0000 7fd073603700 1 -- 172.21.15.69:0/3419490590 shutdown_connections 2023-07-12T15:24:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.839+0000 7fd073603700 1 --2- 172.21.15.69:0/3419490590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd06c104ae0 0x7fd06c104eb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.839+0000 7fd073603700 1 -- 172.21.15.69:0/3419490590 >> 172.21.15.69:0/3419490590 conn(0x7fd06c0fa600 msgr2=0x7fd06c0fca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.839+0000 7fd073603700 1 -- 172.21.15.69:0/3419490590 shutdown_connections 2023-07-12T15:24:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.840+0000 7fd073603700 1 -- 172.21.15.69:0/3419490590 wait complete. 2023-07-12T15:24:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.840+0000 7fd073603700 1 Processor -- start 2023-07-12T15:24:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.840+0000 7fd073603700 1 -- start start 2023-07-12T15:24:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.840+0000 7fd073603700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd06c104ae0 0x7fd06c1936c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:03.844 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.840+0000 7fd073603700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c193c00 con 0x7fd06c104ae0 2023-07-12T15:24:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.841+0000 7fd07139f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd06c104ae0 0x7fd06c1936c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.841+0000 7fd07139f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd06c104ae0 0x7fd06c1936c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39896/0 (socket says 172.21.15.69:39896) 2023-07-12T15:24:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.841+0000 7fd07139f700 1 -- 172.21.15.69:0/1525995472 learned_addr learned my addr 172.21.15.69:0/1525995472 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:24:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.841+0000 7fd07139f700 1 -- 172.21.15.69:0/1525995472 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0680095f0 con 0x7fd06c104ae0 2023-07-12T15:24:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.841+0000 7fd07139f700 1 --2- 172.21.15.69:0/1525995472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd06c104ae0 0x7fd06c1936c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd068005950 tx=0x7fd068004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:03.845 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.841+0000 7fd0627fc700 1 -- 172.21.15.69:0/1525995472 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd06801d070 con 0x7fd06c104ae0 2023-07-12T15:24:03.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.841+0000 7fd0627fc700 1 -- 172.21.15.69:0/1525995472 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fd06800b750 con 0x7fd06c104ae0 2023-07-12T15:24:03.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.841+0000 7fd073603700 1 -- 172.21.15.69:0/1525995472 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06c193e00 con 0x7fd06c104ae0 2023-07-12T15:24:03.846 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.841+0000 7fd0627fc700 1 -- 172.21.15.69:0/1525995472 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd068003830 con 0x7fd06c104ae0 2023-07-12T15:24:03.847 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.841+0000 7fd073603700 1 -- 172.21.15.69:0/1525995472 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd06c194220 con 0x7fd06c104ae0 2023-07-12T15:24:03.848 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.842+0000 7fd073603700 1 -- 172.21.15.69:0/1525995472 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06c062130 con 0x7fd06c104ae0 2023-07-12T15:24:03.849 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.842+0000 7fd0627fc700 1 -- 172.21.15.69:0/1525995472 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7fd068004390 con 0x7fd06c104ae0 2023-07-12T15:24:03.849 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.842+0000 7fd0627fc700 1 --2- 172.21.15.69:0/1525995472 >> v2:172.21.15.69:6800/284433772 conn(0x7fd058038470 0x7fd05803a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:03.849 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.842+0000 7fd0627fc700 1 -- 172.21.15.69:0/1525995472 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fd06804bf20 con 0x7fd06c104ae0 2023-07-12T15:24:03.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.842+0000 7fd070b9e700 1 --2- 172.21.15.69:0/1525995472 >> v2:172.21.15.69:6800/284433772 conn(0x7fd058038470 0x7fd05803a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:03.850 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.843+0000 7fd070b9e700 1 --2- 172.21.15.69:0/1525995472 >> v2:172.21.15.69:6800/284433772 conn(0x7fd058038470 0x7fd05803a920 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd05c006fd0 tx=0x7fd05c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:03.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:03.847+0000 7fd0627fc700 1 -- 172.21.15.69:0/1525995472 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd06800b8c0 con 0x7fd06c104ae0 2023-07-12T15:24:03.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:03 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:04.042 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:04.041+0000 7fd073603700 1 -- 172.21.15.69:0/1525995472 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd06c196fc0 con 0x7fd058038470 2023-07-12T15:24:04.043 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:04.042+0000 7fd0627fc700 1 -- 172.21.15.69:0/1525995472 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7fd06c196fc0 con 0x7fd058038470 2023-07-12T15:24:04.043 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:24:04.044 INFO:teuthology.orchestra.run.smithi069.stdout:[{"addr": "172.21.15.69", "hostname": "smithi069", "labels": [], "status": ""}, {"addr": "172.21.15.145", "hostname": "smithi145", "labels": [], "status": ""}] 2023-07-12T15:24:04.045 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:04.044+0000 7fd073603700 1 -- 172.21.15.69:0/1525995472 >> v2:172.21.15.69:6800/284433772 conn(0x7fd058038470 msgr2=0x7fd05803a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:04.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:04.044+0000 7fd073603700 1 --2- 172.21.15.69:0/1525995472 >> v2:172.21.15.69:6800/284433772 conn(0x7fd058038470 0x7fd05803a920 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd05c006fd0 tx=0x7fd05c006e40 comp rx=0 tx=0).stop 2023-07-12T15:24:04.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:04.044+0000 7fd073603700 1 -- 172.21.15.69:0/1525995472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd06c104ae0 msgr2=0x7fd06c1936c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:04.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:04.044+0000 7fd073603700 1 --2- 172.21.15.69:0/1525995472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd06c104ae0 0x7fd06c1936c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd068005950 tx=0x7fd068004f20 comp rx=0 tx=0).stop 2023-07-12T15:24:04.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:04.044+0000 7fd073603700 1 -- 172.21.15.69:0/1525995472 shutdown_connections 2023-07-12T15:24:04.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:04.044+0000 7fd073603700 1 --2- 172.21.15.69:0/1525995472 >> v2:172.21.15.69:6800/284433772 conn(0x7fd058038470 0x7fd05803a920 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:04.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:04.044+0000 7fd073603700 1 --2- 172.21.15.69:0/1525995472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd06c104ae0 0x7fd06c1936c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:04.046 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:04.044+0000 7fd073603700 1 -- 172.21.15.69:0/1525995472 >> 172.21.15.69:0/1525995472 conn(0x7fd06c0fa600 msgr2=0x7fd06c107020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:04.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:04.044+0000 7fd073603700 1 -- 172.21.15.69:0/1525995472 shutdown_connections 2023-07-12T15:24:04.047 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:04.044+0000 7fd073603700 1 -- 172.21.15.69:0/1525995472 wait complete. 2023-07-12T15:24:04.385 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-12T15:24:04.386 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd crush tunables default 2023-07-12T15:24:04.936 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:24:04.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:04 smithi069 ceph-mon[94241]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:04.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:04 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:05.699 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:05 smithi069 ceph-mon[94241]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-12T15:24:06.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.088+0000 7f8d703b5700 1 -- 172.21.15.69:0/3838286083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d68104b50 msgr2=0x7f8d68104f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:06.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.088+0000 7f8d703b5700 1 --2- 172.21.15.69:0/3838286083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d68104b50 0x7f8d68104f20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009910 tx=0x7f8d64009c20 comp rx=0 tx=0).stop 2023-07-12T15:24:06.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.088+0000 7f8d703b5700 1 -- 172.21.15.69:0/3838286083 shutdown_connections 2023-07-12T15:24:06.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.088+0000 7f8d703b5700 1 --2- 172.21.15.69:0/3838286083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d68104b50 0x7f8d68104f20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:06.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.088+0000 7f8d703b5700 1 -- 172.21.15.69:0/3838286083 >> 172.21.15.69:0/3838286083 conn(0x7f8d680748a0 msgr2=0x7f8d68074ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:06.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.088+0000 7f8d703b5700 1 -- 172.21.15.69:0/3838286083 shutdown_connections 2023-07-12T15:24:06.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.088+0000 7f8d703b5700 1 -- 172.21.15.69:0/3838286083 wait complete. 2023-07-12T15:24:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.088+0000 7f8d703b5700 1 Processor -- start 2023-07-12T15:24:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.088+0000 7f8d703b5700 1 -- start start 2023-07-12T15:24:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.089+0000 7f8d703b5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d68104b50 0x7f8d68194700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.089+0000 7f8d703b5700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d68194c40 con 0x7f8d68104b50 2023-07-12T15:24:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.089+0000 7f8d6e151700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d68104b50 0x7f8d68194700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.089+0000 7f8d6e151700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d68104b50 0x7f8d68194700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39902/0 (socket says 172.21.15.69:39902) 2023-07-12T15:24:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.089+0000 7f8d6e151700 1 -- 172.21.15.69:0/713484036 learned_addr learned my addr 172.21.15.69:0/713484036 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:24:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.089+0000 7f8d6e151700 1 -- 172.21.15.69:0/713484036 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d640095f0 con 0x7f8d68104b50 2023-07-12T15:24:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.089+0000 7f8d6e151700 1 --2- 172.21.15.69:0/713484036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d68104b50 0x7f8d68194700 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8d64005950 tx=0x7f8d64004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.089+0000 7f8d5f7fe700 1 -- 172.21.15.69:0/713484036 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d6401d070 con 0x7f8d68104b50 2023-07-12T15:24:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.089+0000 7f8d5f7fe700 1 -- 172.21.15.69:0/713484036 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f8d6400b750 con 0x7f8d68104b50 2023-07-12T15:24:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.089+0000 7f8d703b5700 1 -- 172.21.15.69:0/713484036 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d68194ea0 con 0x7f8d68104b50 2023-07-12T15:24:06.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.089+0000 7f8d5f7fe700 1 -- 172.21.15.69:0/713484036 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d64003830 con 0x7f8d68104b50 2023-07-12T15:24:06.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.089+0000 7f8d703b5700 1 -- 172.21.15.69:0/713484036 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d68195320 con 0x7f8d68104b50 2023-07-12T15:24:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.090+0000 7f8d5f7fe700 1 -- 172.21.15.69:0/713484036 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7f8d64003af0 con 0x7f8d68104b50 2023-07-12T15:24:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.090+0000 7f8d5f7fe700 1 --2- 172.21.15.69:0/713484036 >> v2:172.21.15.69:6800/284433772 conn(0x7f8d54038420 0x7f8d5403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.090+0000 7f8d5f7fe700 1 -- 172.21.15.69:0/713484036 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f8d6404bf40 con 0x7f8d68104b50 2023-07-12T15:24:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.090+0000 7f8d6d950700 1 --2- 172.21.15.69:0/713484036 >> v2:172.21.15.69:6800/284433772 conn(0x7f8d54038420 0x7f8d5403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:06.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.090+0000 7f8d703b5700 1 -- 172.21.15.69:0/713484036 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d68062130 con 0x7f8d68104b50 2023-07-12T15:24:06.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.091+0000 7f8d6d950700 1 --2- 172.21.15.69:0/713484036 >> v2:172.21.15.69:6800/284433772 conn(0x7f8d54038420 0x7f8d5403a8d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8d58006fd0 tx=0x7f8d58006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:06.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.095+0000 7f8d5f7fe700 1 -- 172.21.15.69:0/713484036 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8d64026080 con 0x7f8d68104b50 2023-07-12T15:24:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.287+0000 7f8d703b5700 1 -- 172.21.15.69:0/713484036 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f8d68198000 con 0x7f8d68104b50 2023-07-12T15:24:06.908 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.906+0000 7f8d5f7fe700 1 -- 172.21.15.69:0/713484036 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8d64003e40 con 0x7f8d68104b50 2023-07-12T15:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.909+0000 7f8d703b5700 1 -- 172.21.15.69:0/713484036 >> v2:172.21.15.69:6800/284433772 conn(0x7f8d54038420 msgr2=0x7f8d5403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.909+0000 7f8d703b5700 1 --2- 172.21.15.69:0/713484036 >> v2:172.21.15.69:6800/284433772 conn(0x7f8d54038420 0x7f8d5403a8d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8d58006fd0 tx=0x7f8d58006e40 comp rx=0 tx=0).stop 2023-07-12T15:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.909+0000 7f8d703b5700 1 -- 172.21.15.69:0/713484036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d68104b50 msgr2=0x7f8d68194700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.909+0000 7f8d703b5700 1 --2- 172.21.15.69:0/713484036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d68104b50 0x7f8d68194700 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8d64005950 tx=0x7f8d64004f20 comp rx=0 tx=0).stop 2023-07-12T15:24:06.910 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.909+0000 7f8d703b5700 1 -- 172.21.15.69:0/713484036 shutdown_connections 2023-07-12T15:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.909+0000 7f8d703b5700 1 --2- 172.21.15.69:0/713484036 >> v2:172.21.15.69:6800/284433772 conn(0x7f8d54038420 0x7f8d5403a8d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.909+0000 7f8d703b5700 1 --2- 172.21.15.69:0/713484036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d68104b50 0x7f8d68194700 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.909+0000 7f8d703b5700 1 -- 172.21.15.69:0/713484036 >> 172.21.15.69:0/713484036 conn(0x7f8d680748a0 msgr2=0x7f8d68068590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.909+0000 7f8d703b5700 1 -- 172.21.15.69:0/713484036 shutdown_connections 2023-07-12T15:24:06.911 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:06.909+0000 7f8d703b5700 1 -- 172.21.15.69:0/713484036 wait complete. 2023-07-12T15:24:06.912 INFO:teuthology.orchestra.run.smithi069.stderr:adjusted tunables profile to default 2023-07-12T15:24:06.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:06 smithi069 ceph-mon[94241]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:06.938 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/713484036' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-12T15:24:07.294 INFO:tasks.cephadm:Adding mon.a on smithi069 2023-07-12T15:24:07.295 INFO:tasks.cephadm:Adding mon.c on smithi069 2023-07-12T15:24:07.295 INFO:tasks.cephadm:Adding mon.b on smithi145 2023-07-12T15:24:07.295 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch apply mon '3;smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi145:172.21.15.145=b' 2023-07-12T15:24:07.871 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:24:07.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:07 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:07 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:07 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:07 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:07 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi145", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:24:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:07 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:07 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:07 smithi069 ceph-mon[94241]: Updating smithi145:/etc/ceph/ceph.conf 2023-07-12T15:24:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/713484036' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-12T15:24:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:07 smithi069 ceph-mon[94241]: osdmap e4: 0 total, 0 up, 0 in 2023-07-12T15:24:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:07 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:07 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:07 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:08.982 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.979+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/3285673817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bac0ffe40 msgr2=0x7f7bac100210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:08.982 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.979+0000 7f7bb1a5f700 1 --2- 172.21.15.145:0/3285673817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bac0ffe40 0x7f7bac100210 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0009910 tx=0x7f7ba0009c20 comp rx=0 tx=0).stop 2023-07-12T15:24:08.982 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.979+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/3285673817 shutdown_connections 2023-07-12T15:24:08.982 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.979+0000 7f7bb1a5f700 1 --2- 172.21.15.145:0/3285673817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bac0ffe40 0x7f7bac100210 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:08.983 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.979+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/3285673817 >> 172.21.15.145:0/3285673817 conn(0x7f7bac0fb6e0 msgr2=0x7f7bac0fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:08.983 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.979+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/3285673817 shutdown_connections 2023-07-12T15:24:08.983 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.979+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/3285673817 wait complete. 2023-07-12T15:24:08.984 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.980+0000 7f7bb1a5f700 1 Processor -- start 2023-07-12T15:24:08.985 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.980+0000 7f7bb1a5f700 1 -- start start 2023-07-12T15:24:08.985 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.980+0000 7f7bb1a5f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bac0ffe40 0x7f7bac19a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:08.985 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.980+0000 7f7bb1a5f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac19aae0 con 0x7f7bac0ffe40 2023-07-12T15:24:08.985 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.980+0000 7f7baaffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bac0ffe40 0x7f7bac19a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:08.985 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.980+0000 7f7baaffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bac0ffe40 0x7f7bac19a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.145:45866/0 (socket says 172.21.15.145:45866) 2023-07-12T15:24:08.985 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.980+0000 7f7baaffd700 1 -- 172.21.15.145:0/4128076758 learned_addr learned my addr 172.21.15.145:0/4128076758 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:24:08.986 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.981+0000 7f7baaffd700 1 -- 172.21.15.145:0/4128076758 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba00095f0 con 0x7f7bac0ffe40 2023-07-12T15:24:08.986 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.981+0000 7f7baaffd700 1 --2- 172.21.15.145:0/4128076758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bac0ffe40 0x7f7bac19a5a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0004f40 tx=0x7f7ba0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:08.986 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.981+0000 7f7bb0a5d700 1 -- 172.21.15.145:0/4128076758 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ba001d070 con 0x7f7bac0ffe40 2023-07-12T15:24:08.986 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.981+0000 7f7bb0a5d700 1 -- 172.21.15.145:0/4128076758 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f7ba000b750 con 0x7f7bac0ffe40 2023-07-12T15:24:08.987 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.981+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/4128076758 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bac19ace0 con 0x7f7bac0ffe40 2023-07-12T15:24:08.987 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.981+0000 7f7bb0a5d700 1 -- 172.21.15.145:0/4128076758 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ba00038c0 con 0x7f7bac0ffe40 2023-07-12T15:24:08.987 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.981+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/4128076758 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bac19b100 con 0x7f7bac0ffe40 2023-07-12T15:24:08.987 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.982+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/4128076758 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bac062130 con 0x7f7bac0ffe40 2023-07-12T15:24:08.987 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.982+0000 7f7bb0a5d700 1 -- 172.21.15.145:0/4128076758 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7f7ba0003b80 con 0x7f7bac0ffe40 2023-07-12T15:24:08.987 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.982+0000 7f7bb0a5d700 1 --2- 172.21.15.145:0/4128076758 >> v2:172.21.15.69:6800/284433772 conn(0x7f7b94038420 0x7f7b9403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:08.988 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.982+0000 7f7bb0a5d700 1 -- 172.21.15.145:0/4128076758 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f7ba004c390 con 0x7f7bac0ffe40 2023-07-12T15:24:08.988 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.982+0000 7f7baa7fc700 1 --2- 172.21.15.145:0/4128076758 >> v2:172.21.15.69:6800/284433772 conn(0x7f7b94038420 0x7f7b9403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:08.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:08 smithi069 ceph-mon[94241]: Updating smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:24:08.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:08 smithi069 ceph-mon[94241]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2023-07-12T15:24:08.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:08 smithi069 ceph-mon[94241]: Updating smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.client.admin.keyring 2023-07-12T15:24:08.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:08 smithi069 ceph-mon[94241]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:08.989 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.983+0000 7f7baa7fc700 1 --2- 172.21.15.145:0/4128076758 >> v2:172.21.15.69:6800/284433772 conn(0x7f7b94038420 0x7f7b9403a8d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c006fd0 tx=0x7f7b9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:08.990 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:08.988+0000 7f7bb0a5d700 1 -- 172.21.15.145:0/4128076758 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7ba002a580 con 0x7f7bac0ffe40 2023-07-12T15:24:09.184 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:09.181+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/4128076758 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi145:172.21.15.145=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bac1946f0 con 0x7f7b94038420 2023-07-12T15:24:09.191 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:09.189+0000 7f7bb0a5d700 1 -- 172.21.15.145:0/4128076758 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7bac1946f0 con 0x7f7b94038420 2023-07-12T15:24:09.191 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled mon update... 2023-07-12T15:24:09.194 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:09.191+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/4128076758 >> v2:172.21.15.69:6800/284433772 conn(0x7f7b94038420 msgr2=0x7f7b9403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:09.195 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:09.191+0000 7f7bb1a5f700 1 --2- 172.21.15.145:0/4128076758 >> v2:172.21.15.69:6800/284433772 conn(0x7f7b94038420 0x7f7b9403a8d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c006fd0 tx=0x7f7b9c006e40 comp rx=0 tx=0).stop 2023-07-12T15:24:09.195 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:09.191+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/4128076758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bac0ffe40 msgr2=0x7f7bac19a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:09.195 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:09.191+0000 7f7bb1a5f700 1 --2- 172.21.15.145:0/4128076758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bac0ffe40 0x7f7bac19a5a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0004f40 tx=0x7f7ba0005020 comp rx=0 tx=0).stop 2023-07-12T15:24:09.195 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:09.191+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/4128076758 shutdown_connections 2023-07-12T15:24:09.195 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:09.191+0000 7f7bb1a5f700 1 --2- 172.21.15.145:0/4128076758 >> v2:172.21.15.69:6800/284433772 conn(0x7f7b94038420 0x7f7b9403a8d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:09.195 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:09.191+0000 7f7bb1a5f700 1 --2- 172.21.15.145:0/4128076758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bac0ffe40 0x7f7bac19a5a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:09.196 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:09.191+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/4128076758 >> 172.21.15.145:0/4128076758 conn(0x7f7bac0fb6e0 msgr2=0x7f7bac104020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:09.196 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:09.191+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/4128076758 shutdown_connections 2023-07-12T15:24:09.196 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:09.191+0000 7f7bb1a5f700 1 -- 172.21.15.145:0/4128076758 wait complete. 2023-07-12T15:24:09.650 DEBUG:teuthology.orchestra.run.smithi069:mon.c> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.c.service 2023-07-12T15:24:09.655 DEBUG:teuthology.orchestra.run.smithi145:mon.b> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.b.service 2023-07-12T15:24:09.659 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-12T15:24:09.660 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph mon dump -f json 2023-07-12T15:24:09.688 INFO:journalctl@ceph.mon.c.smithi069.stdout:-- Logs begin at Wed 2023-07-12 15:05:12 UTC. -- 2023-07-12T15:24:09.689 INFO:journalctl@ceph.mon.b.smithi145.stdout:-- Logs begin at Wed 2023-07-12 15:04:57 UTC. -- 2023-07-12T15:24:10.312 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:24:10.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:10 smithi069 ceph-mon[94241]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi145:172.21.15.145=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:24:10.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:10 smithi069 ceph-mon[94241]: Saving service mon spec with placement smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi145:172.21.15.145=b;count:3 2023-07-12T15:24:10.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:10 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:10.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:10 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:10.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:10 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:10 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:10 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:10 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:10 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:10 smithi069 ceph-mon[94241]: Deploying daemon mon.b on smithi145 2023-07-12T15:24:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:10 smithi069 ceph-mon[94241]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:11.966 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.963+0000 7ff25ac30700 1 -- 172.21.15.145:0/313807321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff254100ce0 msgr2=0x7ff2541010b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:11.966 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.963+0000 7ff25ac30700 1 --2- 172.21.15.145:0/313807321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff254100ce0 0x7ff2541010b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff248009910 tx=0x7ff248009c20 comp rx=0 tx=0).stop 2023-07-12T15:24:11.966 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.963+0000 7ff25ac30700 1 -- 172.21.15.145:0/313807321 shutdown_connections 2023-07-12T15:24:11.967 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.963+0000 7ff25ac30700 1 --2- 172.21.15.145:0/313807321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff254100ce0 0x7ff2541010b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:11.967 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.963+0000 7ff25ac30700 1 -- 172.21.15.145:0/313807321 >> 172.21.15.145:0/313807321 conn(0x7ff2540748a0 msgr2=0x7ff254074ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:11.967 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.963+0000 7ff25ac30700 1 -- 172.21.15.145:0/313807321 shutdown_connections 2023-07-12T15:24:11.967 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.963+0000 7ff25ac30700 1 -- 172.21.15.145:0/313807321 wait complete. 2023-07-12T15:24:11.967 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.964+0000 7ff25ac30700 1 Processor -- start 2023-07-12T15:24:11.967 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.964+0000 7ff25ac30700 1 -- start start 2023-07-12T15:24:11.967 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.964+0000 7ff25ac30700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff254100ce0 0x7ff25419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:11.968 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.964+0000 7ff25ac30700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25419ab40 con 0x7ff254100ce0 2023-07-12T15:24:11.968 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.965+0000 7ff2589cc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff254100ce0 0x7ff25419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:11.968 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.965+0000 7ff2589cc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff254100ce0 0x7ff25419a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.145:45872/0 (socket says 172.21.15.145:45872) 2023-07-12T15:24:11.968 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.965+0000 7ff2589cc700 1 -- 172.21.15.145:0/816375733 learned_addr learned my addr 172.21.15.145:0/816375733 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:24:11.969 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.965+0000 7ff2589cc700 1 -- 172.21.15.145:0/816375733 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2480095f0 con 0x7ff254100ce0 2023-07-12T15:24:11.970 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.965+0000 7ff2589cc700 1 --2- 172.21.15.145:0/816375733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff254100ce0 0x7ff25419a600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff248004f40 tx=0x7ff248005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:11.970 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.965+0000 7ff251ffb700 1 -- 172.21.15.145:0/816375733 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff24801d070 con 0x7ff254100ce0 2023-07-12T15:24:11.970 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.965+0000 7ff25ac30700 1 -- 172.21.15.145:0/816375733 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25419ad40 con 0x7ff254100ce0 2023-07-12T15:24:11.970 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.965+0000 7ff251ffb700 1 -- 172.21.15.145:0/816375733 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7ff248004370 con 0x7ff254100ce0 2023-07-12T15:24:11.970 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.965+0000 7ff251ffb700 1 -- 172.21.15.145:0/816375733 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff248003990 con 0x7ff254100ce0 2023-07-12T15:24:11.970 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.965+0000 7ff25ac30700 1 -- 172.21.15.145:0/816375733 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25419b160 con 0x7ff254100ce0 2023-07-12T15:24:11.970 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.966+0000 7ff251ffb700 1 -- 172.21.15.145:0/816375733 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7ff248003c00 con 0x7ff254100ce0 2023-07-12T15:24:11.971 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.966+0000 7ff251ffb700 1 --2- 172.21.15.145:0/816375733 >> v2:172.21.15.69:6800/284433772 conn(0x7ff244038420 0x7ff24403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:11.971 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.967+0000 7ff251ffb700 1 -- 172.21.15.145:0/816375733 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7ff24804c090 con 0x7ff254100ce0 2023-07-12T15:24:11.971 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.967+0000 7ff25ac30700 1 -- 172.21.15.145:0/816375733 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff254062130 con 0x7ff254100ce0 2023-07-12T15:24:11.971 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.967+0000 7ff253fff700 1 --2- 172.21.15.145:0/816375733 >> v2:172.21.15.69:6800/284433772 conn(0x7ff244038420 0x7ff24403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:11.971 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.967+0000 7ff253fff700 1 --2- 172.21.15.145:0/816375733 >> v2:172.21.15.69:6800/284433772 conn(0x7ff244038420 0x7ff24403a8d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff240006fd0 tx=0x7ff240006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:11.973 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:11.972+0000 7ff251ffb700 1 -- 172.21.15.145:0/816375733 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff248026080 con 0x7ff254100ce0 2023-07-12T15:24:12.269 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:12.267+0000 7ff25ac30700 1 -- 172.21.15.145:0/816375733 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff254195b90 con 0x7ff254100ce0 2023-07-12T15:24:12.270 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:12.268+0000 7ff251ffb700 1 -- 172.21.15.145:0/816375733 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7ff248029950 con 0x7ff254100ce0 2023-07-12T15:24:12.270 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:24:12.271 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":1,"fsid":"dc75a490-20c7-11ee-9b31-001a4aab830c","modified":"2023-07-12T15:22:46.541643Z","created":"2023-07-12T15:22:46.541643Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-12T15:24:12.272 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:12.270+0000 7ff25ac30700 1 -- 172.21.15.145:0/816375733 >> v2:172.21.15.69:6800/284433772 conn(0x7ff244038420 msgr2=0x7ff24403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:12.273 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:12.270+0000 7ff25ac30700 1 --2- 172.21.15.145:0/816375733 >> v2:172.21.15.69:6800/284433772 conn(0x7ff244038420 0x7ff24403a8d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff240006fd0 tx=0x7ff240006e40 comp rx=0 tx=0).stop 2023-07-12T15:24:12.273 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:12.270+0000 7ff25ac30700 1 -- 172.21.15.145:0/816375733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff254100ce0 msgr2=0x7ff25419a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:12.273 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:12.270+0000 7ff25ac30700 1 --2- 172.21.15.145:0/816375733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff254100ce0 0x7ff25419a600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff248004f40 tx=0x7ff248005020 comp rx=0 tx=0).stop 2023-07-12T15:24:12.273 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:12.270+0000 7ff25ac30700 1 -- 172.21.15.145:0/816375733 shutdown_connections 2023-07-12T15:24:12.274 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:12.270+0000 7ff25ac30700 1 --2- 172.21.15.145:0/816375733 >> v2:172.21.15.69:6800/284433772 conn(0x7ff244038420 0x7ff24403a8d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:12.274 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:12.270+0000 7ff25ac30700 1 --2- 172.21.15.145:0/816375733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff254100ce0 0x7ff25419a600 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:12.274 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:12.270+0000 7ff25ac30700 1 -- 172.21.15.145:0/816375733 >> 172.21.15.145:0/816375733 conn(0x7ff2540748a0 msgr2=0x7ff2541045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:12.275 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:12.270+0000 7ff25ac30700 1 -- 172.21.15.145:0/816375733 shutdown_connections 2023-07-12T15:24:12.275 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:12.270+0000 7ff25ac30700 1 -- 172.21.15.145:0/816375733 wait complete. 2023-07-12T15:24:12.275 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 1 2023-07-12T15:24:12.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:12 smithi069 ceph-mon[94241]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:12.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/816375733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-12T15:24:13.186 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:13 smithi145 ceph-mon[95801]: mon.b@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-12T15:24:13.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:13 smithi145 systemd[1]: Started Ceph mon.b for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:24:13.646 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-12T15:24:13.647 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph mon dump -f json 2023-07-12T15:24:14.202 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:24:18.411 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.409+0000 7f1fbe72c700 1 -- 172.21.15.145:0/876316739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fa00035c0 msgr2=0x7f1fa0005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:18.411 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.409+0000 7f1fbe72c700 1 --2- 172.21.15.145:0/876316739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fa00035c0 0x7f1fa0005a40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8068bd0 tx=0x7f1fa800feb0 comp rx=0 tx=0).stop 2023-07-12T15:24:18.412 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.409+0000 7f1fbe72c700 1 -- 172.21.15.145:0/876316739 shutdown_connections 2023-07-12T15:24:18.412 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.409+0000 7f1fbe72c700 1 --2- 172.21.15.145:0/876316739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fa00035c0 0x7f1fa0005a40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:18.412 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.409+0000 7f1fbe72c700 1 -- 172.21.15.145:0/876316739 >> 172.21.15.145:0/876316739 conn(0x7f1fb80fa720 msgr2=0x7f1fb80fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:18.412 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.409+0000 7f1fbe72c700 1 -- 172.21.15.145:0/876316739 shutdown_connections 2023-07-12T15:24:18.412 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.409+0000 7f1fbe72c700 1 -- 172.21.15.145:0/876316739 wait complete. 2023-07-12T15:24:18.414 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.410+0000 7f1fbe72c700 1 Processor -- start 2023-07-12T15:24:18.415 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.410+0000 7f1fbe72c700 1 -- start start 2023-07-12T15:24:18.415 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fbe72c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa00035c0 0x7f1fb819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:18.415 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fbe72c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fb819ac50 0x7f1fb8194790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:18.415 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fbe72c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb819b1d0 con 0x7f1fb819ac50 2023-07-12T15:24:18.416 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fbe72c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb819b310 con 0x7f1fa00035c0 2023-07-12T15:24:18.416 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fb7fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa00035c0 0x7f1fb819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:18.416 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fb7fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa00035c0 0x7f1fb819a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:43740/0 (socket says 172.21.15.145:43740) 2023-07-12T15:24:18.417 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fb7fff700 1 -- 172.21.15.145:0/775118376 learned_addr learned my addr 172.21.15.145:0/775118376 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:24:18.417 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fb77fe700 1 --2- 172.21.15.145:0/775118376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fb819ac50 0x7f1fb8194790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:18.417 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fb7fff700 1 -- 172.21.15.145:0/775118376 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa00035c0 msgr2=0x7f1fb819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-12T15:24:18.417 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fb7fff700 1 -- 172.21.15.145:0/775118376 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa00035c0 msgr2=0x7f1fb819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-12T15:24:18.418 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fb7fff700 1 --2- 172.21.15.145:0/775118376 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa00035c0 0x7f1fb819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-12T15:24:18.419 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fb7fff700 1 --2- 172.21.15.145:0/775118376 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa00035c0 0x7f1fb819a710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-12T15:24:18.419 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fb77fe700 1 -- 172.21.15.145:0/775118376 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa00035c0 msgr2=0x7f1fb819a710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-12T15:24:18.419 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.411+0000 7f1fb77fe700 1 --2- 172.21.15.145:0/775118376 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa00035c0 0x7f1fb819a710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:18.419 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.412+0000 7f1fb77fe700 1 -- 172.21.15.145:0/775118376 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa800f900 con 0x7f1fb819ac50 2023-07-12T15:24:18.419 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.412+0000 7f1fb77fe700 1 --2- 172.21.15.145:0/775118376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fb819ac50 0x7f1fb8194790 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1fac00b4d0 tx=0x7f1fac00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:18.420 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.412+0000 7f1fb57fa700 1 -- 172.21.15.145:0/775118376 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f1fac011630 con 0x7f1fb819ac50 2023-07-12T15:24:18.420 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.412+0000 7f1fb57fa700 1 -- 172.21.15.145:0/775118376 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f1fac011c20 con 0x7f1fb819ac50 2023-07-12T15:24:18.420 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.412+0000 7f1fbe72c700 1 -- 172.21.15.145:0/775118376 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb8194d90 con 0x7f1fb819ac50 2023-07-12T15:24:18.420 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.412+0000 7f1fb57fa700 1 -- 172.21.15.145:0/775118376 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f1fac010b80 con 0x7f1fb819ac50 2023-07-12T15:24:18.420 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.412+0000 7f1fbe72c700 1 -- 172.21.15.145:0/775118376 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb81952e0 con 0x7f1fb819ac50 2023-07-12T15:24:18.421 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.413+0000 7f1fb57fa700 1 -- 172.21.15.145:0/775118376 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7f1fac01a410 con 0x7f1fb819ac50 2023-07-12T15:24:18.421 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.413+0000 7f1fb57fa700 1 --2- 172.21.15.145:0/775118376 >> v2:172.21.15.69:6800/284433772 conn(0x7f1fa4038270 0x7f1fa403a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:18.421 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.413+0000 7f1fb57fa700 1 -- 172.21.15.145:0/775118376 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f1fac04a2a0 con 0x7f1fb819ac50 2023-07-12T15:24:18.422 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.413+0000 7f1fbe72c700 1 -- 172.21.15.145:0/775118376 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb8195610 con 0x7f1fb819ac50 2023-07-12T15:24:18.422 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.414+0000 7f1fb7fff700 1 --2- 172.21.15.145:0/775118376 >> v2:172.21.15.69:6800/284433772 conn(0x7f1fa4038270 0x7f1fa403a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:18.422 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.414+0000 7f1fb7fff700 1 --2- 172.21.15.145:0/775118376 >> v2:172.21.15.69:6800/284433772 conn(0x7f1fa4038270 0x7f1fa403a720 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8005140 tx=0x7f1fa8004e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:18.422 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.419+0000 7f1fb57fa700 1 -- 172.21.15.145:0/775118376 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1fac010df0 con 0x7f1fb819ac50 2023-07-12T15:24:18.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:24:18.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:18.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: mon.a calling monitor election 2023-07-12T15:24:18.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:18.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:18.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:18.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: mon.b calling monitor election 2023-07-12T15:24:18.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2023-07-12T15:24:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: monmap e2: 2 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0]} removed_ranks: {} 2023-07-12T15:24:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: fsmap 2023-07-12T15:24:18.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: osdmap e4: 0 total, 0 up, 0 in 2023-07-12T15:24:18.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: mgrmap e13: y(active, since 34s) 2023-07-12T15:24:18.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: overall HEALTH_OK 2023-07-12T15:24:18.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:18.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:18.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:24:18.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:18.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: mon.a calling monitor election 2023-07-12T15:24:18.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:18.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:18.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:18.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: mon.b calling monitor election 2023-07-12T15:24:18.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:18.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:18.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:18.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:18.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:18.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2023-07-12T15:24:18.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: monmap e2: 2 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0]} removed_ranks: {} 2023-07-12T15:24:18.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: fsmap 2023-07-12T15:24:18.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: osdmap e4: 0 total, 0 up, 0 in 2023-07-12T15:24:18.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: mgrmap e13: y(active, since 34s) 2023-07-12T15:24:18.606 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: overall HEALTH_OK 2023-07-12T15:24:18.606 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:18.606 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:18.686 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.684+0000 7f1fbe72c700 1 -- 172.21.15.145:0/775118376 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fb8062130 con 0x7f1fb819ac50 2023-07-12T15:24:18.686 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.685+0000 7f1fb57fa700 1 -- 172.21.15.145:0/775118376 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f1fac047020 con 0x7f1fb819ac50 2023-07-12T15:24:18.687 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:24:18.687 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":2,"fsid":"dc75a490-20c7-11ee-9b31-001a4aab830c","modified":"2023-07-12T15:24:13.120654Z","created":"2023-07-12T15:22:46.541643Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3300","nonce":0},{"type":"v1","addr":"172.21.15.145:6789","nonce":0}]},"addr":"172.21.15.145:6789/0","public_addr":"172.21.15.145:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-12T15:24:18.688 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.687+0000 7f1fbe72c700 1 -- 172.21.15.145:0/775118376 >> v2:172.21.15.69:6800/284433772 conn(0x7f1fa4038270 msgr2=0x7f1fa403a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:18.688 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.687+0000 7f1fbe72c700 1 --2- 172.21.15.145:0/775118376 >> v2:172.21.15.69:6800/284433772 conn(0x7f1fa4038270 0x7f1fa403a720 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8005140 tx=0x7f1fa8004e50 comp rx=0 tx=0).stop 2023-07-12T15:24:18.689 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.687+0000 7f1fbe72c700 1 -- 172.21.15.145:0/775118376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fb819ac50 msgr2=0x7f1fb8194790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:18.689 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.687+0000 7f1fbe72c700 1 --2- 172.21.15.145:0/775118376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fb819ac50 0x7f1fb8194790 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1fac00b4d0 tx=0x7f1fac00b890 comp rx=0 tx=0).stop 2023-07-12T15:24:18.689 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.687+0000 7f1fbe72c700 1 -- 172.21.15.145:0/775118376 shutdown_connections 2023-07-12T15:24:18.689 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.687+0000 7f1fbe72c700 1 --2- 172.21.15.145:0/775118376 >> v2:172.21.15.69:6800/284433772 conn(0x7f1fa4038270 0x7f1fa403a720 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:18.690 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.687+0000 7f1fbe72c700 1 --2- 172.21.15.145:0/775118376 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa00035c0 0x7f1fb819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:18.691 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.687+0000 7f1fbe72c700 1 --2- 172.21.15.145:0/775118376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fb819ac50 0x7f1fb8194790 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:18.691 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.687+0000 7f1fbe72c700 1 -- 172.21.15.145:0/775118376 >> 172.21.15.145:0/775118376 conn(0x7f1fb80fa720 msgr2=0x7f1fb8101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:18.691 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.687+0000 7f1fbe72c700 1 -- 172.21.15.145:0/775118376 shutdown_connections 2023-07-12T15:24:18.691 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:18.687+0000 7f1fbe72c700 1 -- 172.21.15.145:0/775118376 wait complete. 2023-07-12T15:24:18.691 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 2 2023-07-12T15:24:19.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:19 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:19 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:19 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:19 smithi145 ceph-mon[95801]: Deploying daemon mon.c on smithi069 2023-07-12T15:24:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/775118376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-12T15:24:19.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:19 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:19.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:19 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:19 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:19 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:19 smithi069 ceph-mon[94241]: Deploying daemon mon.c on smithi069 2023-07-12T15:24:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/775118376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-12T15:24:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:19 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:20.065 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-12T15:24:20.066 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph mon dump -f json 2023-07-12T15:24:20.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:20 smithi069 ceph-mon[94241]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:20.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:20 smithi145 ceph-mon[95801]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:20.628 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:24:20.832 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:20 smithi069 systemd[1]: Starting Ceph mon.c for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:24:21.088 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:20 smithi069 podman[102737]: 2023-07-12T15:24:21.088 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 bash[102737]: 5fc942fdad3501c04cdd7d1bce589a9917a34cfa906fcdebf2cac4696615f013 2023-07-12T15:24:21.088 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: set uid:gid to 167:167 (ceph:ceph) 2023-07-12T15:24:21.089 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: ceph version 18.1.2-55-gb1bd0540 (b1bd054042b7797ba5b70d145a1da31693dda494) reef (rc), process ceph-mon, pid 7 2023-07-12T15:24:21.089 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: pidfile_write: ignore empty --pid-file 2023-07-12T15:24:21.089 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: load: jerasure load: lrc 2023-07-12T15:24:21.089 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: RocksDB version: 7.9.2 2023-07-12T15:24:21.089 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Git sha 0 2023-07-12T15:24:21.089 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Compile date 2023-07-12 12:18:34 2023-07-12T15:24:21.089 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: DB SUMMARY 2023-07-12T15:24:21.089 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: DB Session ID: HXTSK61D7ODLJ9YCHNW5 2023-07-12T15:24:21.090 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: CURRENT file: CURRENT 2023-07-12T15:24:21.090 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: IDENTITY file: IDENTITY 2023-07-12T15:24:21.090 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-12T15:24:21.090 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-12T15:24:21.090 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2023-07-12T15:24:21.090 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.error_if_exists: 0 2023-07-12T15:24:21.090 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.create_if_missing: 0 2023-07-12T15:24:21.091 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.paranoid_checks: 1 2023-07-12T15:24:21.091 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-12T15:24:21.091 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-12T15:24:21.091 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-12T15:24:21.091 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.env: 0x557c4a634720 2023-07-12T15:24:21.091 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.fs: PosixFileSystem 2023-07-12T15:24:21.091 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.info_log: 0x557c4d57b3e0 2023-07-12T15:24:21.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_file_opening_threads: 16 2023-07-12T15:24:21.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.statistics: (nil) 2023-07-12T15:24:21.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.use_fsync: 0 2023-07-12T15:24:21.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_log_file_size: 0 2023-07-12T15:24:21.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-12T15:24:21.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-12T15:24:21.092 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.keep_log_file_num: 1000 2023-07-12T15:24:21.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.recycle_log_file_num: 0 2023-07-12T15:24:21.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.allow_fallocate: 1 2023-07-12T15:24:21.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.allow_mmap_reads: 0 2023-07-12T15:24:21.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.allow_mmap_writes: 0 2023-07-12T15:24:21.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.use_direct_reads: 0 2023-07-12T15:24:21.093 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-12T15:24:21.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.create_missing_column_families: 0 2023-07-12T15:24:21.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.db_log_dir: 2023-07-12T15:24:21.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.wal_dir: 2023-07-12T15:24:21.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-12T15:24:21.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-12T15:24:21.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-12T15:24:21.094 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-12T15:24:21.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-12T15:24:21.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-12T15:24:21.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.advise_random_on_open: 1 2023-07-12T15:24:21.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.db_write_buffer_size: 0 2023-07-12T15:24:21.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.write_buffer_manager: 0x557c4c80c5a0 2023-07-12T15:24:21.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-12T15:24:21.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-12T15:24:21.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-12T15:24:21.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.rate_limiter: (nil) 2023-07-12T15:24:21.096 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-12T15:24:21.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.wal_recovery_mode: 2 2023-07-12T15:24:21.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.enable_thread_tracking: 0 2023-07-12T15:24:21.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.enable_pipelined_write: 0 2023-07-12T15:24:21.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.unordered_write: 0 2023-07-12T15:24:21.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-12T15:24:21.097 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-12T15:24:21.098 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-12T15:24:21.098 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-12T15:24:21.098 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.row_cache: None 2023-07-12T15:24:21.098 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.wal_filter: None 2023-07-12T15:24:21.098 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-12T15:24:21.098 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.allow_ingest_behind: 0 2023-07-12T15:24:21.098 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.two_write_queues: 0 2023-07-12T15:24:21.099 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.manual_wal_flush: 0 2023-07-12T15:24:21.099 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.wal_compression: 0 2023-07-12T15:24:21.099 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.atomic_flush: 0 2023-07-12T15:24:21.099 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-12T15:24:21.099 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-12T15:24:21.100 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-12T15:24:21.100 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.log_readahead_size: 0 2023-07-12T15:24:21.100 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-12T15:24:21.100 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.best_efforts_recovery: 0 2023-07-12T15:24:21.100 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-12T15:24:21.100 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-12T15:24:21.100 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.allow_data_in_errors: 0 2023-07-12T15:24:21.104 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.db_host_id: __hostname__ 2023-07-12T15:24:21.105 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-12T15:24:21.105 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_background_jobs: 2 2023-07-12T15:24:21.105 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_background_compactions: -1 2023-07-12T15:24:21.106 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_subcompactions: 1 2023-07-12T15:24:21.106 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-12T15:24:21.106 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-12T15:24:21.106 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-12T15:24:21.107 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_total_wal_size: 0 2023-07-12T15:24:21.107 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-12T15:24:21.107 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-12T15:24:21.108 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-12T15:24:21.108 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-12T15:24:21.108 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_open_files: -1 2023-07-12T15:24:21.108 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.bytes_per_sync: 0 2023-07-12T15:24:21.109 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-12T15:24:21.109 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-12T15:24:21.109 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compaction_readahead_size: 0 2023-07-12T15:24:21.111 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_background_flushes: -1 2023-07-12T15:24:21.111 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Compression algorithms supported: 2023-07-12T15:24:21.111 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-12T15:24:21.111 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: kZSTD supported: 0 2023-07-12T15:24:21.111 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: kXpressCompression supported: 0 2023-07-12T15:24:21.112 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: kLZ4HCCompression supported: 1 2023-07-12T15:24:21.112 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: kZlibCompression supported: 1 2023-07-12T15:24:21.112 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: kSnappyCompression supported: 1 2023-07-12T15:24:21.112 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: kLZ4Compression supported: 1 2023-07-12T15:24:21.112 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: kBZip2Compression supported: 0 2023-07-12T15:24:21.112 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-12T15:24:21.112 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-12T15:24:21.112 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-12T15:24:21.113 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-12T15:24:21.113 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-12T15:24:21.113 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.merge_operator: 2023-07-12T15:24:21.113 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compaction_filter: None 2023-07-12T15:24:21.113 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compaction_filter_factory: None 2023-07-12T15:24:21.113 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.sst_partitioner_factory: None 2023-07-12T15:24:21.113 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-12T15:24:21.114 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-12T15:24:21.114 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557c4d57b500) 2023-07-12T15:24:21.114 INFO:journalctl@ceph.mon.c.smithi069.stdout: cache_index_and_filter_blocks: 1 2023-07-12T15:24:21.114 INFO:journalctl@ceph.mon.c.smithi069.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-12T15:24:21.114 INFO:journalctl@ceph.mon.c.smithi069.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-12T15:24:21.114 INFO:journalctl@ceph.mon.c.smithi069.stdout: pin_top_level_index_and_filter: 1 2023-07-12T15:24:21.114 INFO:journalctl@ceph.mon.c.smithi069.stdout: index_type: 0 2023-07-12T15:24:21.115 INFO:journalctl@ceph.mon.c.smithi069.stdout: data_block_index_type: 0 2023-07-12T15:24:21.115 INFO:journalctl@ceph.mon.c.smithi069.stdout: index_shortening: 1 2023-07-12T15:24:21.115 INFO:journalctl@ceph.mon.c.smithi069.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-12T15:24:21.115 INFO:journalctl@ceph.mon.c.smithi069.stdout: checksum: 4 2023-07-12T15:24:21.115 INFO:journalctl@ceph.mon.c.smithi069.stdout: no_block_cache: 0 2023-07-12T15:24:21.115 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_cache: 0x557c4c88d1f0 2023-07-12T15:24:21.115 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_cache_name: BinnedLRUCache 2023-07-12T15:24:21.115 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_cache_options: 2023-07-12T15:24:21.116 INFO:journalctl@ceph.mon.c.smithi069.stdout: capacity : 536870912 2023-07-12T15:24:21.116 INFO:journalctl@ceph.mon.c.smithi069.stdout: num_shard_bits : 4 2023-07-12T15:24:21.118 INFO:journalctl@ceph.mon.c.smithi069.stdout: strict_capacity_limit : 0 2023-07-12T15:24:21.119 INFO:journalctl@ceph.mon.c.smithi069.stdout: high_pri_pool_ratio: 0.000 2023-07-12T15:24:21.119 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_cache_compressed: (nil) 2023-07-12T15:24:21.119 INFO:journalctl@ceph.mon.c.smithi069.stdout: persistent_cache: (nil) 2023-07-12T15:24:21.119 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_size: 4096 2023-07-12T15:24:21.119 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_size_deviation: 10 2023-07-12T15:24:21.119 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_restart_interval: 16 2023-07-12T15:24:21.119 INFO:journalctl@ceph.mon.c.smithi069.stdout: index_block_restart_interval: 1 2023-07-12T15:24:21.119 INFO:journalctl@ceph.mon.c.smithi069.stdout: metadata_block_size: 4096 2023-07-12T15:24:21.120 INFO:journalctl@ceph.mon.c.smithi069.stdout: partition_filters: 0 2023-07-12T15:24:21.120 INFO:journalctl@ceph.mon.c.smithi069.stdout: use_delta_encoding: 1 2023-07-12T15:24:21.120 INFO:journalctl@ceph.mon.c.smithi069.stdout: filter_policy: bloomfilter 2023-07-12T15:24:21.120 INFO:journalctl@ceph.mon.c.smithi069.stdout: whole_key_filtering: 1 2023-07-12T15:24:21.120 INFO:journalctl@ceph.mon.c.smithi069.stdout: verify_compression: 0 2023-07-12T15:24:21.120 INFO:journalctl@ceph.mon.c.smithi069.stdout: read_amp_bytes_per_bit: 0 2023-07-12T15:24:21.120 INFO:journalctl@ceph.mon.c.smithi069.stdout: format_version: 5 2023-07-12T15:24:21.121 INFO:journalctl@ceph.mon.c.smithi069.stdout: enable_index_compression: 1 2023-07-12T15:24:21.121 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_align: 0 2023-07-12T15:24:21.121 INFO:journalctl@ceph.mon.c.smithi069.stdout: max_auto_readahead_size: 262144 2023-07-12T15:24:21.121 INFO:journalctl@ceph.mon.c.smithi069.stdout: prepopulate_block_cache: 0 2023-07-12T15:24:21.121 INFO:journalctl@ceph.mon.c.smithi069.stdout: initial_auto_readahead_size: 8192 2023-07-12T15:24:21.121 INFO:journalctl@ceph.mon.c.smithi069.stdout: num_file_reads_for_auto_readahead: 2 2023-07-12T15:24:21.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.write_buffer_size: 33554432 2023-07-12T15:24:21.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_write_buffer_number: 2 2023-07-12T15:24:21.122 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compression: NoCompression 2023-07-12T15:24:21.122 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.bottommost_compression: Disabled 2023-07-12T15:24:21.122 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.prefix_extractor: nullptr 2023-07-12T15:24:21.122 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-12T15:24:21.122 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.num_levels: 7 2023-07-12T15:24:21.122 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-12T15:24:21.122 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-12T15:24:21.123 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-12T15:24:21.123 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-12T15:24:21.123 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-12T15:24:21.123 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-12T15:24:21.123 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-12T15:24:21.123 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-12T15:24:21.123 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-12T15:24:21.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-12T15:24:21.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-12T15:24:21.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-12T15:24:21.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-12T15:24:21.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compression_opts.level: 32767 2023-07-12T15:24:21.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compression_opts.strategy: 0 2023-07-12T15:24:21.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-12T15:24:21.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-12T15:24:21.125 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-12T15:24:21.125 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-12T15:24:21.125 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compression_opts.enabled: false 2023-07-12T15:24:21.125 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-12T15:24:21.125 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-12T15:24:21.125 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-12T15:24:21.125 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-12T15:24:21.126 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.target_file_size_base: 67108864 2023-07-12T15:24:21.126 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-12T15:24:21.126 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-12T15:24:21.126 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-12T15:24:21.126 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-12T15:24:21.126 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-12T15:24:21.126 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-12T15:24:21.126 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-12T15:24:21.127 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-12T15:24:21.127 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-12T15:24:21.127 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-12T15:24:21.127 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-12T15:24:21.127 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-12T15:24:21.127 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-12T15:24:21.127 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-12T15:24:21.127 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.arena_block_size: 1048576 2023-07-12T15:24:21.128 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-12T15:24:21.128 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-12T15:24:21.128 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.disable_auto_compactions: 0 2023-07-12T15:24:21.128 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-12T15:24:21.128 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-12T15:24:21.129 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-12T15:24:21.129 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-12T15:24:21.129 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-12T15:24:21.129 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-12T15:24:21.129 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-12T15:24:21.129 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-12T15:24:21.129 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-12T15:24:21.129 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-12T15:24:21.130 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.table_properties_collectors: 2023-07-12T15:24:21.130 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.inplace_update_support: 0 2023-07-12T15:24:21.130 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-12T15:24:21.130 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-12T15:24:21.130 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-12T15:24:21.130 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-12T15:24:21.130 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.bloom_locality: 0 2023-07-12T15:24:21.131 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.max_successive_merges: 0 2023-07-12T15:24:21.131 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-12T15:24:21.131 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.paranoid_file_checks: 0 2023-07-12T15:24:21.131 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.force_consistency_checks: 1 2023-07-12T15:24:21.131 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.report_bg_io_stats: 0 2023-07-12T15:24:21.131 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.ttl: 2592000 2023-07-12T15:24:21.131 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-12T15:24:21.131 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-12T15:24:21.132 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-12T15:24:21.132 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.enable_blob_files: false 2023-07-12T15:24:21.132 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.min_blob_size: 0 2023-07-12T15:24:21.132 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.blob_file_size: 268435456 2023-07-12T15:24:21.132 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-12T15:24:21.132 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-12T15:24:21.132 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-12T15:24:21.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-12T15:24:21.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-12T15:24:21.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.blob_file_starting_level: 0 2023-07-12T15:24:21.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-12T15:24:21.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-12T15:24:21.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-12T15:24:21.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4b4ac8bf-6b76-43be-af4e-0544f46b8c93 2023-07-12T15:24:21.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689175461071589, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-12T15:24:21.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-12T15:24:21.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689175461071977, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689175461, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4b4ac8bf-6b76-43be-af4e-0544f46b8c93", "db_session_id": "HXTSK61D7ODLJ9YCHNW5", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-12T15:24:21.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689175461072043, "job": 1, "event": "recovery_finished"} 2023-07-12T15:24:21.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-12T15:24:21.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-12T15:24:21.134 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557c4c928000 2023-07-12T15:24:21.135 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: DB pointer 0x557c4c914000 2023-07-12T15:24:21.135 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-12T15:24:21.135 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-12T15:24:21.135 INFO:journalctl@ceph.mon.c.smithi069.stdout: ** DB Stats ** 2023-07-12T15:24:21.135 INFO:journalctl@ceph.mon.c.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-12T15:24:21.135 INFO:journalctl@ceph.mon.c.smithi069.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-12T15:24:21.135 INFO:journalctl@ceph.mon.c.smithi069.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-12T15:24:21.135 INFO:journalctl@ceph.mon.c.smithi069.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-12T15:24:21.136 INFO:journalctl@ceph.mon.c.smithi069.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-12T15:24:21.136 INFO:journalctl@ceph.mon.c.smithi069.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-12T15:24:21.136 INFO:journalctl@ceph.mon.c.smithi069.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-12T15:24:21.136 INFO:journalctl@ceph.mon.c.smithi069.stdout: 2023-07-12T15:24:21.136 INFO:journalctl@ceph.mon.c.smithi069.stdout: ** Compaction Stats [default] ** 2023-07-12T15:24:21.136 INFO:journalctl@ceph.mon.c.smithi069.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-12T15:24:21.136 INFO:journalctl@ceph.mon.c.smithi069.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-12T15:24:21.136 INFO:journalctl@ceph.mon.c.smithi069.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-12T15:24:21.137 INFO:journalctl@ceph.mon.c.smithi069.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-12T15:24:21.137 INFO:journalctl@ceph.mon.c.smithi069.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-12T15:24:21.137 INFO:journalctl@ceph.mon.c.smithi069.stdout: 2023-07-12T15:24:21.137 INFO:journalctl@ceph.mon.c.smithi069.stdout: ** Compaction Stats [default] ** 2023-07-12T15:24:21.137 INFO:journalctl@ceph.mon.c.smithi069.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-12T15:24:21.137 INFO:journalctl@ceph.mon.c.smithi069.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-12T15:24:21.137 INFO:journalctl@ceph.mon.c.smithi069.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-12T15:24:21.138 INFO:journalctl@ceph.mon.c.smithi069.stdout: 2023-07-12T15:24:21.138 INFO:journalctl@ceph.mon.c.smithi069.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-12T15:24:21.138 INFO:journalctl@ceph.mon.c.smithi069.stdout: 2023-07-12T15:24:21.138 INFO:journalctl@ceph.mon.c.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-12T15:24:21.138 INFO:journalctl@ceph.mon.c.smithi069.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-12T15:24:21.138 INFO:journalctl@ceph.mon.c.smithi069.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-12T15:24:21.138 INFO:journalctl@ceph.mon.c.smithi069.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-12T15:24:21.138 INFO:journalctl@ceph.mon.c.smithi069.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-12T15:24:21.139 INFO:journalctl@ceph.mon.c.smithi069.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-12T15:24:21.139 INFO:journalctl@ceph.mon.c.smithi069.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-12T15:24:21.139 INFO:journalctl@ceph.mon.c.smithi069.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-12T15:24:21.139 INFO:journalctl@ceph.mon.c.smithi069.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-12T15:24:21.139 INFO:journalctl@ceph.mon.c.smithi069.stdout: Block cache BinnedLRUCache@0x557c4c88d1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-07-12T15:24:21.139 INFO:journalctl@ceph.mon.c.smithi069.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-12T15:24:21.139 INFO:journalctl@ceph.mon.c.smithi069.stdout: 2023-07-12T15:24:21.139 INFO:journalctl@ceph.mon.c.smithi069.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-12T15:24:21.140 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-12T15:24:21.140 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: using public_addrv [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-07-12T15:24:21.140 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: starting mon.c rank -1 at public addrs [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] at bind addrs [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:24:21.140 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(???) e0 preinit fsid dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:24:21.140 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing).mds e1 new map 2023-07-12T15:24:21.140 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-12T15:24:21.140 INFO:journalctl@ceph.mon.c.smithi069.stdout: e1 2023-07-12T15:24:21.141 INFO:journalctl@ceph.mon.c.smithi069.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-12T15:24:21.141 INFO:journalctl@ceph.mon.c.smithi069.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-12T15:24:21.141 INFO:journalctl@ceph.mon.c.smithi069.stdout: legacy client fscid: -1 2023-07-12T15:24:21.141 INFO:journalctl@ceph.mon.c.smithi069.stdout: 2023-07-12T15:24:21.141 INFO:journalctl@ceph.mon.c.smithi069.stdout: No filesystems configured 2023-07-12T15:24:21.141 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-12T15:24:21.141 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-12T15:24:21.141 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-12T15:24:21.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-12T15:24:21.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-12T15:24:21.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-12T15:24:21.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-12T15:24:21.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-12T15:24:21.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-12T15:24:21.142 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-12T15:24:21.143 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:21.143 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:21.143 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:21.143 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:21.143 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi145", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:24:21.143 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:21.143 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:21.143 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: Updating smithi145:/etc/ceph/ceph.conf 2023-07-12T15:24:21.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/713484036' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-12T15:24:21.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: osdmap e4: 0 total, 0 up, 0 in 2023-07-12T15:24:21.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:21.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:21.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:21.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: Updating smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:24:21.144 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2023-07-12T15:24:21.145 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: Updating smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.client.admin.keyring 2023-07-12T15:24:21.145 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:21.145 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi145:172.21.15.145=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:24:21.145 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: Saving service mon spec with placement smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi145:172.21.15.145=b;count:3 2023-07-12T15:24:21.145 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:21.145 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:21.145 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:21.145 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:21.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:21.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:21.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:21.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: Deploying daemon mon.b on smithi145 2023-07-12T15:24:21.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:21.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:21.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/816375733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-12T15:24:21.146 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:24:21.147 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:21.147 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.a calling monitor election 2023-07-12T15:24:21.147 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:21.147 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:21.147 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:21.147 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.b calling monitor election 2023-07-12T15:24:21.147 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:21.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:21.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:21.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:21.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:21.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2023-07-12T15:24:21.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: monmap e2: 2 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0]} removed_ranks: {} 2023-07-12T15:24:21.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: fsmap 2023-07-12T15:24:21.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: osdmap e4: 0 total, 0 up, 0 in 2023-07-12T15:24:21.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mgrmap e13: y(active, since 34s) 2023-07-12T15:24:21.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: overall HEALTH_OK 2023-07-12T15:24:21.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:21.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:21.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:21.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:21.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:21.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: Deploying daemon mon.c on smithi069 2023-07-12T15:24:21.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/775118376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-12T15:24:21.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:21.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:21.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-12T15:24:21.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: expand_channel_meta expand map: {default=false} 2023-07-12T15:24:21.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: expand_channel_meta from 'false' to 'false' 2023-07-12T15:24:21.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: expand_channel_meta expanded map: {default=false} 2023-07-12T15:24:21.151 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: expand_channel_meta expand map: {default=info} 2023-07-12T15:24:21.151 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: expand_channel_meta from 'info' to 'info' 2023-07-12T15:24:21.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: expand_channel_meta expanded map: {default=info} 2023-07-12T15:24:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: expand_channel_meta expand map: {default=daemon} 2023-07-12T15:24:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-12T15:24:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: expand_channel_meta expanded map: {default=daemon} 2023-07-12T15:24:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: expand_channel_meta expand map: {default=debug} 2023-07-12T15:24:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: expand_channel_meta from 'debug' to 'debug' 2023-07-12T15:24:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: expand_channel_meta expanded map: {default=debug} 2023-07-12T15:24:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 ceph-mon[102777]: mon.c@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-12T15:24:21.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:21 smithi069 systemd[1]: Started Ceph mon.c for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:24:26.327 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.326+0000 7f182e0f6700 1 -- 172.21.15.145:0/1736019313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18140035c0 msgr2=0x7f1814005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:26.328 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.326+0000 7f182e0f6700 1 --2- 172.21.15.145:0/1736019313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18140035c0 0x7f1814005a40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f18280fec20 tx=0x7f181800fe60 comp rx=0 tx=0).stop 2023-07-12T15:24:26.328 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.326+0000 7f182e0f6700 1 -- 172.21.15.145:0/1736019313 shutdown_connections 2023-07-12T15:24:26.328 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.326+0000 7f182e0f6700 1 --2- 172.21.15.145:0/1736019313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18140035c0 0x7f1814005a40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:26.328 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.326+0000 7f182e0f6700 1 -- 172.21.15.145:0/1736019313 >> 172.21.15.145:0/1736019313 conn(0x7f18280749c0 msgr2=0x7f1828074dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:26.328 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.326+0000 7f182e0f6700 1 -- 172.21.15.145:0/1736019313 shutdown_connections 2023-07-12T15:24:26.329 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.326+0000 7f182e0f6700 1 -- 172.21.15.145:0/1736019313 wait complete. 2023-07-12T15:24:26.330 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f182e0f6700 1 Processor -- start 2023-07-12T15:24:26.330 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f182e0f6700 1 -- start start 2023-07-12T15:24:26.331 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f182e0f6700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18140035c0 0x7f1828100910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:26.331 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f182e0f6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1828100e50 0x7f18281012c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:26.331 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f182e0f6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18281a0bf0 0x7f18281a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:26.331 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f182e0f6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18281a3700 con 0x7f1828100e50 2023-07-12T15:24:26.332 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f182e0f6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18281a3870 con 0x7f18281a0bf0 2023-07-12T15:24:26.332 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f182e0f6700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18281a3a30 con 0x7f18140035c0 2023-07-12T15:24:26.332 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f1827fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18281a0bf0 0x7f18281a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:26.332 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f1826ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1828100e50 0x7f18281012c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:26.332 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f1827fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18281a0bf0 0x7f18281a2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:43768/0 (socket says 172.21.15.145:43768) 2023-07-12T15:24:26.332 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f1827fff700 1 -- 172.21.15.145:0/3346077227 learned_addr learned my addr 172.21.15.145:0/3346077227 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:24:26.333 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f1826ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1828100e50 0x7f18281012c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.145:45928/0 (socket says 172.21.15.145:45928) 2023-07-12T15:24:26.333 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.327+0000 7f18277fe700 1 --2- 172.21.15.145:0/3346077227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18140035c0 0x7f1828100910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:26.333 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f1827fff700 1 -- 172.21.15.145:0/3346077227 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18281a0bf0 msgr2=0x7f18281a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-07-12T15:24:26.333 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f1827fff700 1 -- 172.21.15.145:0/3346077227 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18281a0bf0 msgr2=0x7f18281a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-12T15:24:26.333 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f1827fff700 1 --2- 172.21.15.145:0/3346077227 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18281a0bf0 0x7f18281a2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-12T15:24:26.333 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f18277fe700 1 -- 172.21.15.145:0/3346077227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18140035c0 msgr2=0x7f1828100910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-12T15:24:26.333 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f18277fe700 1 -- 172.21.15.145:0/3346077227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18140035c0 msgr2=0x7f1828100910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-12T15:24:26.334 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f18277fe700 1 --2- 172.21.15.145:0/3346077227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18140035c0 0x7f1828100910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-12T15:24:26.334 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f18277fe700 1 --2- 172.21.15.145:0/3346077227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18140035c0 0x7f1828100910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-12T15:24:26.334 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f1827fff700 1 --2- 172.21.15.145:0/3346077227 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18281a0bf0 0x7f18281a2fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-12T15:24:26.334 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f1826ffd700 1 -- 172.21.15.145:0/3346077227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18140035c0 msgr2=0x7f1828100910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-12T15:24:26.334 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f1826ffd700 1 --2- 172.21.15.145:0/3346077227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18140035c0 0x7f1828100910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:26.335 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f1826ffd700 1 -- 172.21.15.145:0/3346077227 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18281a0bf0 msgr2=0x7f18281a2fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-12T15:24:26.335 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f1826ffd700 1 --2- 172.21.15.145:0/3346077227 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18281a0bf0 0x7f18281a2fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:26.335 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f1826ffd700 1 -- 172.21.15.145:0/3346077227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181800f8b0 con 0x7f1828100e50 2023-07-12T15:24:26.335 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f1826ffd700 1 --2- 172.21.15.145:0/3346077227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1828100e50 0x7f18281012c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f181000b550 tx=0x7f181000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:26.336 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f1824ff9700 1 -- 172.21.15.145:0/3346077227 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810004230 con 0x7f1828100e50 2023-07-12T15:24:26.336 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f1824ff9700 1 -- 172.21.15.145:0/3346077227 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f1810004390 con 0x7f1828100e50 2023-07-12T15:24:26.336 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f182e0f6700 1 -- 172.21.15.145:0/3346077227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18281a3cb0 con 0x7f1828100e50 2023-07-12T15:24:26.336 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f1824ff9700 1 -- 172.21.15.145:0/3346077227 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810011a80 con 0x7f1828100e50 2023-07-12T15:24:26.336 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.328+0000 7f182e0f6700 1 -- 172.21.15.145:0/3346077227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18281a41b0 con 0x7f1828100e50 2023-07-12T15:24:26.337 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.329+0000 7f1824ff9700 1 -- 172.21.15.145:0/3346077227 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7f1810011be0 con 0x7f1828100e50 2023-07-12T15:24:26.337 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.329+0000 7f1824ff9700 1 --2- 172.21.15.145:0/3346077227 >> v2:172.21.15.69:6800/284433772 conn(0x7f1808038900 0x7f180803adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:26.338 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.329+0000 7f182e0f6700 1 -- 172.21.15.145:0/3346077227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182818ede0 con 0x7f1828100e50 2023-07-12T15:24:26.338 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.330+0000 7f1824ff9700 1 -- 172.21.15.145:0/3346077227 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f181004b680 con 0x7f1828100e50 2023-07-12T15:24:26.338 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.330+0000 7f18277fe700 1 --2- 172.21.15.145:0/3346077227 >> v2:172.21.15.69:6800/284433772 conn(0x7f1808038900 0x7f180803adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:26.338 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.330+0000 7f18277fe700 1 --2- 172.21.15.145:0/3346077227 >> v2:172.21.15.69:6800/284433772 conn(0x7f1808038900 0x7f180803adb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f18180039c0 tx=0x7f1818001e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:26.338 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.335+0000 7f1824ff9700 1 -- 172.21.15.145:0/3346077227 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f181001a400 con 0x7f1828100e50 2023-07-12T15:24:26.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:24:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: mon.a calling monitor election 2023-07-12T15:24:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: mon.b calling monitor election 2023-07-12T15:24:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: mon.c calling monitor election 2023-07-12T15:24:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-12T15:24:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: monmap e3: 3 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],c=[v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0]} removed_ranks: {} 2023-07-12T15:24:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: fsmap 2023-07-12T15:24:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: osdmap e4: 0 total, 0 up, 0 in 2023-07-12T15:24:26.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: mgrmap e13: y(active, since 42s) 2023-07-12T15:24:26.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: overall HEALTH_OK 2023-07-12T15:24:26.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:26.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:26.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:24:26.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: mon.a calling monitor election 2023-07-12T15:24:26.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: mon.b calling monitor election 2023-07-12T15:24:26.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:26.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:26.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: mon.c calling monitor election 2023-07-12T15:24:26.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:26.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:26.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-12T15:24:26.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: monmap e3: 3 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],c=[v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0]} removed_ranks: {} 2023-07-12T15:24:26.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: fsmap 2023-07-12T15:24:26.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: osdmap e4: 0 total, 0 up, 0 in 2023-07-12T15:24:26.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: mgrmap e13: y(active, since 42s) 2023-07-12T15:24:26.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: overall HEALTH_OK 2023-07-12T15:24:26.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:26.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:26.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:24:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: mon.a calling monitor election 2023-07-12T15:24:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: mon.b calling monitor election 2023-07-12T15:24:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:24:26.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:26.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: mon.c calling monitor election 2023-07-12T15:24:26.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:26.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:26.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:26.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-12T15:24:26.605 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: monmap e3: 3 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],c=[v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0]} removed_ranks: {} 2023-07-12T15:24:26.605 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: fsmap 2023-07-12T15:24:26.605 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: osdmap e4: 0 total, 0 up, 0 in 2023-07-12T15:24:26.605 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: mgrmap e13: y(active, since 42s) 2023-07-12T15:24:26.606 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: overall HEALTH_OK 2023-07-12T15:24:26.606 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:26.606 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:26.615 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.613+0000 7f182e0f6700 1 -- 172.21.15.145:0/3346077227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1828062130 con 0x7f1828100e50 2023-07-12T15:24:26.616 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.614+0000 7f1824ff9700 1 -- 172.21.15.145:0/3346077227 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1272 (secure 0 0 0) 0x7f1810018350 con 0x7f1828100e50 2023-07-12T15:24:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-07-12T15:24:26.616 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":3,"fsid":"dc75a490-20c7-11ee-9b31-001a4aab830c","modified":"2023-07-12T15:24:21.092131Z","created":"2023-07-12T15:22:46.541643Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3300","nonce":0},{"type":"v1","addr":"172.21.15.145:6789","nonce":0}]},"addr":"172.21.15.145:6789/0","public_addr":"172.21.15.145:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3301","nonce":0},{"type":"v1","addr":"172.21.15.69:6790","nonce":0}]},"addr":"172.21.15.69:6790/0","public_addr":"172.21.15.69:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2023-07-12T15:24:26.618 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.617+0000 7f182e0f6700 1 -- 172.21.15.145:0/3346077227 >> v2:172.21.15.69:6800/284433772 conn(0x7f1808038900 msgr2=0x7f180803adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:26.619 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.617+0000 7f182e0f6700 1 --2- 172.21.15.145:0/3346077227 >> v2:172.21.15.69:6800/284433772 conn(0x7f1808038900 0x7f180803adb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f18180039c0 tx=0x7f1818001e70 comp rx=0 tx=0).stop 2023-07-12T15:24:26.619 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.617+0000 7f182e0f6700 1 -- 172.21.15.145:0/3346077227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1828100e50 msgr2=0x7f18281012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:26.619 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.617+0000 7f182e0f6700 1 --2- 172.21.15.145:0/3346077227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1828100e50 0x7f18281012c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f181000b550 tx=0x7f181000b910 comp rx=0 tx=0).stop 2023-07-12T15:24:26.619 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.617+0000 7f182e0f6700 1 -- 172.21.15.145:0/3346077227 shutdown_connections 2023-07-12T15:24:26.620 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.617+0000 7f182e0f6700 1 --2- 172.21.15.145:0/3346077227 >> v2:172.21.15.69:6800/284433772 conn(0x7f1808038900 0x7f180803adb0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:26.620 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.617+0000 7f182e0f6700 1 --2- 172.21.15.145:0/3346077227 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18281a0bf0 0x7f18281a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:26.620 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.617+0000 7f182e0f6700 1 --2- 172.21.15.145:0/3346077227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18140035c0 0x7f1828100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:26.620 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.617+0000 7f182e0f6700 1 --2- 172.21.15.145:0/3346077227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1828100e50 0x7f18281012c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:26.620 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.617+0000 7f182e0f6700 1 -- 172.21.15.145:0/3346077227 >> 172.21.15.145:0/3346077227 conn(0x7f18280749c0 msgr2=0x7f1828103010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:26.620 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.617+0000 7f182e0f6700 1 -- 172.21.15.145:0/3346077227 shutdown_connections 2023-07-12T15:24:26.620 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:26.617+0000 7f182e0f6700 1 -- 172.21.15.145:0/3346077227 wait complete. 2023-07-12T15:24:26.621 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 3 2023-07-12T15:24:27.080 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-12T15:24:27.081 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph config generate-minimal-conf 2023-07-12T15:24:27.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:27 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:27.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:27 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:27.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:27 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:27.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/3346077227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-12T15:24:27.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:27 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:27.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:27 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:27 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:27 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/3346077227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-12T15:24:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:27 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:27.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:27 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:27.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:27 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:27.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:27 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:27.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/3346077227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-12T15:24:27.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:27 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:24:27.702 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:24:28.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[94241]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:28.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:28.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:28.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:28.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:28.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:28.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:28.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[102777]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:28.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:28.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:28.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:28.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:28.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:28.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:28 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:28.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:28 smithi145 ceph-mon[95801]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:28 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:28 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:28 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:28 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:28 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:28 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:29.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.997+0000 7fdec68bb700 1 -- 172.21.15.69:0/695188958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec00fea40 msgr2=0x7fdec00fee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:29.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.997+0000 7fdec68bb700 1 --2- 172.21.15.69:0/695188958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec00fea40 0x7fdec00fee10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0009910 tx=0x7fdeb0009c20 comp rx=0 tx=0).stop 2023-07-12T15:24:29.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.997+0000 7fdec68bb700 1 -- 172.21.15.69:0/695188958 shutdown_connections 2023-07-12T15:24:29.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.997+0000 7fdec68bb700 1 --2- 172.21.15.69:0/695188958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec00fea40 0x7fdec00fee10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:29.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.997+0000 7fdec68bb700 1 -- 172.21.15.69:0/695188958 >> 172.21.15.69:0/695188958 conn(0x7fdec0075500 msgr2=0x7fdec0075900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:29.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.997+0000 7fdec68bb700 1 -- 172.21.15.69:0/695188958 shutdown_connections 2023-07-12T15:24:29.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.997+0000 7fdec68bb700 1 -- 172.21.15.69:0/695188958 wait complete. 2023-07-12T15:24:29.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.998+0000 7fdec68bb700 1 Processor -- start 2023-07-12T15:24:29.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.998+0000 7fdec68bb700 1 -- start start 2023-07-12T15:24:29.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.998+0000 7fdec68bb700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec00fea40 0x7fdec019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:29.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.998+0000 7fdec68bb700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdec019ad10 0x7fdec0194850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:29.014 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.998+0000 7fdec68bb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0194d90 0x7fdec0195230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:29.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.998+0000 7fdec68bb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec019b320 con 0x7fdec0194d90 2023-07-12T15:24:29.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.998+0000 7fdec68bb700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec01a15f0 con 0x7fdec019ad10 2023-07-12T15:24:29.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdec68bb700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec01a1730 con 0x7fdec00fea40 2023-07-12T15:24:29.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec00fea40 0x7fdec019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:29.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebffff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec00fea40 0x7fdec019a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:33358/0 (socket says 172.21.15.69:33358) 2023-07-12T15:24:29.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebffff700 1 -- 172.21.15.69:0/3904109843 learned_addr learned my addr 172.21.15.69:0/3904109843 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:24:29.015 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdec4e58700 1 --2- 172.21.15.69:0/3904109843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0194d90 0x7fdec0195230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:29.016 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebf7fe700 1 --2- 172.21.15.69:0/3904109843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdec019ad10 0x7fdec0194850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:29.016 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebffff700 1 -- 172.21.15.69:0/3904109843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdec019ad10 msgr2=0x7fdec0194850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:29.016 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebffff700 1 --2- 172.21.15.69:0/3904109843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdec019ad10 0x7fdec0194850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:29.016 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebffff700 1 -- 172.21.15.69:0/3904109843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0194d90 msgr2=0x7fdec0195230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:29.016 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebffff700 1 --2- 172.21.15.69:0/3904109843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0194d90 0x7fdec0195230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:29.016 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebffff700 1 -- 172.21.15.69:0/3904109843 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb00095f0 con 0x7fdec00fea40 2023-07-12T15:24:29.016 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdec4e58700 1 --2- 172.21.15.69:0/3904109843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0194d90 0x7fdec0195230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:24:29.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebf7fe700 1 --2- 172.21.15.69:0/3904109843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdec019ad10 0x7fdec0194850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:24:29.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebffff700 1 --2- 172.21.15.69:0/3904109843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec00fea40 0x7fdec019a7d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000bcf0 tx=0x7fdeb000bd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:29.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebd7fa700 1 -- 172.21.15.69:0/3904109843 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb001d070 con 0x7fdec00fea40 2023-07-12T15:24:29.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebd7fa700 1 -- 172.21.15.69:0/3904109843 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fdeb0003d20 con 0x7fdec00fea40 2023-07-12T15:24:29.017 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdec68bb700 1 -- 172.21.15.69:0/3904109843 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec01a19b0 con 0x7fdec00fea40 2023-07-12T15:24:29.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdebd7fa700 1 -- 172.21.15.69:0/3904109843 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb000fd00 con 0x7fdec00fea40 2023-07-12T15:24:29.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:28.999+0000 7fdec68bb700 1 -- 172.21.15.69:0/3904109843 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec01a1ea0 con 0x7fdec00fea40 2023-07-12T15:24:29.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.000+0000 7fdec68bb700 1 -- 172.21.15.69:0/3904109843 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec0107260 con 0x7fdec00fea40 2023-07-12T15:24:29.018 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.001+0000 7fdebd7fa700 1 -- 172.21.15.69:0/3904109843 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7fdeb000f460 con 0x7fdec00fea40 2023-07-12T15:24:29.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.001+0000 7fdebd7fa700 1 --2- 172.21.15.69:0/3904109843 >> v2:172.21.15.69:6800/284433772 conn(0x7fdea8038900 0x7fdea803adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:29.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.001+0000 7fdebd7fa700 1 -- 172.21.15.69:0/3904109843 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fdeb004c170 con 0x7fdec00fea40 2023-07-12T15:24:29.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.001+0000 7fdebf7fe700 1 --2- 172.21.15.69:0/3904109843 >> v2:172.21.15.69:6800/284433772 conn(0x7fdea8038900 0x7fdea803adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:29.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.002+0000 7fdebf7fe700 1 --2- 172.21.15.69:0/3904109843 >> v2:172.21.15.69:6800/284433772 conn(0x7fdea8038900 0x7fdea803adb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4009910 tx=0x7fdeb4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:29.019 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.007+0000 7fdebd7fa700 1 -- 172.21.15.69:0/3904109843 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdeb000e5c0 con 0x7fdec00fea40 2023-07-12T15:24:29.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.201+0000 7fdec68bb700 1 -- 172.21.15.69:0/3904109843 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fdec0062130 con 0x7fdec00fea40 2023-07-12T15:24:29.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.201+0000 7fdebd7fa700 1 -- 172.21.15.69:0/3904109843 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7fdeb000e3e0 con 0x7fdec00fea40 2023-07-12T15:24:29.204 INFO:teuthology.orchestra.run.smithi069.stdout:# minimal ceph.conf for dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:24:29.204 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2023-07-12T15:24:29.204 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:24:29.204 INFO:teuthology.orchestra.run.smithi069.stdout: mon_host = [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-07-12T15:24:29.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.203+0000 7fdec68bb700 1 -- 172.21.15.69:0/3904109843 >> v2:172.21.15.69:6800/284433772 conn(0x7fdea8038900 msgr2=0x7fdea803adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:29.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.203+0000 7fdec68bb700 1 --2- 172.21.15.69:0/3904109843 >> v2:172.21.15.69:6800/284433772 conn(0x7fdea8038900 0x7fdea803adb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4009910 tx=0x7fdeb4008040 comp rx=0 tx=0).stop 2023-07-12T15:24:29.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.203+0000 7fdec68bb700 1 -- 172.21.15.69:0/3904109843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec00fea40 msgr2=0x7fdec019a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:29.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.203+0000 7fdec68bb700 1 --2- 172.21.15.69:0/3904109843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec00fea40 0x7fdec019a7d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000bcf0 tx=0x7fdeb000bd20 comp rx=0 tx=0).stop 2023-07-12T15:24:29.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.203+0000 7fdec68bb700 1 -- 172.21.15.69:0/3904109843 shutdown_connections 2023-07-12T15:24:29.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.203+0000 7fdec68bb700 1 --2- 172.21.15.69:0/3904109843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0194d90 0x7fdec0195230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:29.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.203+0000 7fdec68bb700 1 --2- 172.21.15.69:0/3904109843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec00fea40 0x7fdec019a7d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:29.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.203+0000 7fdec68bb700 1 --2- 172.21.15.69:0/3904109843 >> v2:172.21.15.69:6800/284433772 conn(0x7fdea8038900 0x7fdea803adb0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:29.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.203+0000 7fdec68bb700 1 --2- 172.21.15.69:0/3904109843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdec019ad10 0x7fdec0194850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:29.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.203+0000 7fdec68bb700 1 -- 172.21.15.69:0/3904109843 >> 172.21.15.69:0/3904109843 conn(0x7fdec0075500 msgr2=0x7fdec01043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:29.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.203+0000 7fdec68bb700 1 -- 172.21.15.69:0/3904109843 shutdown_connections 2023-07-12T15:24:29.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:29.203+0000 7fdec68bb700 1 -- 172.21.15.69:0/3904109843 wait complete. 2023-07-12T15:24:29.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: Updating smithi069:/etc/ceph/ceph.conf 2023-07-12T15:24:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: Updating smithi145:/etc/ceph/ceph.conf 2023-07-12T15:24:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: Updating smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:24:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: Updating smithi069:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:24:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: Reconfiguring mon.a (unknown last config time)... 2023-07-12T15:24:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-12T15:24:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:29.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: Reconfiguring daemon mon.a on smithi069 2023-07-12T15:24:29.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3904109843' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:29.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: Updating smithi069:/etc/ceph/ceph.conf 2023-07-12T15:24:29.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: Updating smithi145:/etc/ceph/ceph.conf 2023-07-12T15:24:29.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: Updating smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:24:29.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: Updating smithi069:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:24:29.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: Reconfiguring mon.a (unknown last config time)... 2023-07-12T15:24:29.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:29.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-12T15:24:29.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:29.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: Reconfiguring daemon mon.a on smithi069 2023-07-12T15:24:29.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3904109843' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:29.848 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-12T15:24:29.849 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:24:29.849 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.conf 2023-07-12T15:24:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: Updating smithi069:/etc/ceph/ceph.conf 2023-07-12T15:24:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: Updating smithi145:/etc/ceph/ceph.conf 2023-07-12T15:24:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: Updating smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:24:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: Updating smithi069:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:24:29.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:29.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: Reconfiguring mon.a (unknown last config time)... 2023-07-12T15:24:29.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:29.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-12T15:24:29.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:29.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: Reconfiguring daemon mon.a on smithi069 2023-07-12T15:24:29.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3904109843' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:29.877 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:24:29.879 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-12T15:24:29.945 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-07-12T15:24:29.945 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.conf 2023-07-12T15:24:29.981 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-07-12T15:24:29.981 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-12T15:24:30.051 INFO:tasks.cephadm:Adding mgr.y on smithi069 2023-07-12T15:24:30.051 INFO:tasks.cephadm:Adding mgr.x on smithi145 2023-07-12T15:24:30.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch apply mgr '2;smithi069=y;smithi145=x' 2023-07-12T15:24:30.640 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:24:30.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:30 smithi145 ceph-mon[95801]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:30.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:30 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:30.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:30 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:30.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:30 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:30.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:30 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-12T15:24:30.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:30 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:30.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:30 smithi069 ceph-mon[102777]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:30.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:30 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:30.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:30 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:30.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:30 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:30.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:30 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-12T15:24:30.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:30 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:30.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:30 smithi069 ceph-mon[94241]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:30.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:30 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:30.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:30 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:30.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:30 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:30.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:30 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-12T15:24:30.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:30 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:31.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:31 smithi145 ceph-mon[95801]: Reconfiguring mon.c (monmap changed)... 2023-07-12T15:24:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:31 smithi145 ceph-mon[95801]: Reconfiguring daemon mon.c on smithi069 2023-07-12T15:24:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:31 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:31 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:31 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:31 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-12T15:24:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:31 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:31.856 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.854+0000 7f247af28700 1 -- 172.21.15.145:0/3132737843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24741001e0 msgr2=0x7f24741005f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:31.856 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.854+0000 7f247af28700 1 --2- 172.21.15.145:0/3132737843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24741001e0 0x7f24741005f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2468009910 tx=0x7f2468009c20 comp rx=0 tx=0).stop 2023-07-12T15:24:31.856 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.854+0000 7f247af28700 1 -- 172.21.15.145:0/3132737843 shutdown_connections 2023-07-12T15:24:31.857 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.854+0000 7f247af28700 1 --2- 172.21.15.145:0/3132737843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24741001e0 0x7f24741005f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:31.857 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.854+0000 7f247af28700 1 -- 172.21.15.145:0/3132737843 >> 172.21.15.145:0/3132737843 conn(0x7f24740f9910 msgr2=0x7f24740fbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:31.857 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.854+0000 7f247af28700 1 -- 172.21.15.145:0/3132737843 shutdown_connections 2023-07-12T15:24:31.857 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.854+0000 7f247af28700 1 -- 172.21.15.145:0/3132737843 wait complete. 2023-07-12T15:24:31.859 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.855+0000 7f247af28700 1 Processor -- start 2023-07-12T15:24:31.859 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.855+0000 7f247af28700 1 -- start start 2023-07-12T15:24:31.859 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.855+0000 7f247af28700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24741001e0 0x7f24740fe940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:31.860 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.855+0000 7f247af28700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24741974e0 0x7f24741998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:31.860 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.855+0000 7f247af28700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2474199ea0 0x7f247419c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:31.860 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.855+0000 7f247af28700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247419cac0 con 0x7f2474199ea0 2023-07-12T15:24:31.860 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.855+0000 7f247af28700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247419cc30 con 0x7f24741974e0 2023-07-12T15:24:31.860 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.855+0000 7f247af28700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247419cdf0 con 0x7f24741001e0 2023-07-12T15:24:31.860 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.855+0000 7f2478cc4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24741001e0 0x7f24740fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:31.861 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.855+0000 7f24794c5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2474199ea0 0x7f247419c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:31.861 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.855+0000 7f2478cc4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24741001e0 0x7f24740fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.145:58708/0 (socket says 172.21.15.145:58708) 2023-07-12T15:24:31.861 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.855+0000 7f2478cc4700 1 -- 172.21.15.145:0/1769516935 learned_addr learned my addr 172.21.15.145:0/1769516935 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:24:31.861 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.856+0000 7f2473fff700 1 --2- 172.21.15.145:0/1769516935 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24741974e0 0x7f24741998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:31.861 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.856+0000 7f24794c5700 1 -- 172.21.15.145:0/1769516935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24741001e0 msgr2=0x7f24740fe940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:31.861 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.856+0000 7f24794c5700 1 --2- 172.21.15.145:0/1769516935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24741001e0 0x7f24740fe940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:31.861 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.856+0000 7f24794c5700 1 -- 172.21.15.145:0/1769516935 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24741974e0 msgr2=0x7f24741998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:31.861 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.856+0000 7f24794c5700 1 --2- 172.21.15.145:0/1769516935 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24741974e0 0x7f24741998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:31.862 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.856+0000 7f24794c5700 1 -- 172.21.15.145:0/1769516935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24680095f0 con 0x7f2474199ea0 2023-07-12T15:24:31.862 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.856+0000 7f24794c5700 1 --2- 172.21.15.145:0/1769516935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2474199ea0 0x7f247419c320 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f246c00bd50 tx=0x7f246c00bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:31.862 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.856+0000 7f2471ffb700 1 -- 172.21.15.145:0/1769516935 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c00cbb0 con 0x7f2474199ea0 2023-07-12T15:24:31.862 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.856+0000 7f2471ffb700 1 -- 172.21.15.145:0/1769516935 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f246c00cd10 con 0x7f2474199ea0 2023-07-12T15:24:31.862 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.856+0000 7f247af28700 1 -- 172.21.15.145:0/1769516935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247419d0d0 con 0x7f2474199ea0 2023-07-12T15:24:31.862 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.856+0000 7f2471ffb700 1 -- 172.21.15.145:0/1769516935 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c0125a0 con 0x7f2474199ea0 2023-07-12T15:24:31.863 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.856+0000 7f247af28700 1 -- 172.21.15.145:0/1769516935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247419d650 con 0x7f2474199ea0 2023-07-12T15:24:31.864 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.857+0000 7f2471ffb700 1 -- 172.21.15.145:0/1769516935 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7f246c014440 con 0x7f2474199ea0 2023-07-12T15:24:31.865 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.857+0000 7f2471ffb700 1 --2- 172.21.15.145:0/1769516935 >> v2:172.21.15.69:6800/284433772 conn(0x7f2464038900 0x7f246403adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:31.865 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.857+0000 7f2471ffb700 1 -- 172.21.15.145:0/1769516935 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f246c04a6d0 con 0x7f2474199ea0 2023-07-12T15:24:31.867 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.857+0000 7f247af28700 1 -- 172.21.15.145:0/1769516935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247418ce30 con 0x7f2474199ea0 2023-07-12T15:24:31.868 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.858+0000 7f2478cc4700 1 --2- 172.21.15.145:0/1769516935 >> v2:172.21.15.69:6800/284433772 conn(0x7f2464038900 0x7f246403adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:31.869 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.858+0000 7f2478cc4700 1 --2- 172.21.15.145:0/1769516935 >> v2:172.21.15.69:6800/284433772 conn(0x7f2464038900 0x7f246403adb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2468005190 tx=0x7f246801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:31.869 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:31.863+0000 7f2471ffb700 1 -- 172.21.15.145:0/1769516935 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f246c01b890 con 0x7f2474199ea0 2023-07-12T15:24:31.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[94241]: Reconfiguring mon.c (monmap changed)... 2023-07-12T15:24:31.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[94241]: Reconfiguring daemon mon.c on smithi069 2023-07-12T15:24:31.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:31.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:31.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:31.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-12T15:24:31.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:31.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[102777]: Reconfiguring mon.c (monmap changed)... 2023-07-12T15:24:31.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[102777]: Reconfiguring daemon mon.c on smithi069 2023-07-12T15:24:31.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:31.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:31.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-12T15:24:31.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-12T15:24:31.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:31 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:32.073 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.072+0000 7f247af28700 1 -- 172.21.15.145:0/1769516935 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi069=y;smithi145=x", "target": ["mon-mgr", ""]}) v1 -- 0x7f2474060f40 con 0x7f2464038900 2023-07-12T15:24:32.079 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.078+0000 7f2471ffb700 1 -- 172.21.15.145:0/1769516935 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2474060f40 con 0x7f2464038900 2023-07-12T15:24:32.080 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled mgr update... 2023-07-12T15:24:32.082 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.080+0000 7f247af28700 1 -- 172.21.15.145:0/1769516935 >> v2:172.21.15.69:6800/284433772 conn(0x7f2464038900 msgr2=0x7f246403adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:32.082 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.080+0000 7f247af28700 1 --2- 172.21.15.145:0/1769516935 >> v2:172.21.15.69:6800/284433772 conn(0x7f2464038900 0x7f246403adb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2468005190 tx=0x7f246801a040 comp rx=0 tx=0).stop 2023-07-12T15:24:32.083 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.080+0000 7f247af28700 1 -- 172.21.15.145:0/1769516935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2474199ea0 msgr2=0x7f247419c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:32.083 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.080+0000 7f247af28700 1 --2- 172.21.15.145:0/1769516935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2474199ea0 0x7f247419c320 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f246c00bd50 tx=0x7f246c00bd80 comp rx=0 tx=0).stop 2023-07-12T15:24:32.083 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.080+0000 7f247af28700 1 -- 172.21.15.145:0/1769516935 shutdown_connections 2023-07-12T15:24:32.083 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.080+0000 7f247af28700 1 --2- 172.21.15.145:0/1769516935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2474199ea0 0x7f247419c320 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:32.083 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.080+0000 7f247af28700 1 --2- 172.21.15.145:0/1769516935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24741001e0 0x7f24740fe940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:32.083 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.080+0000 7f247af28700 1 --2- 172.21.15.145:0/1769516935 >> v2:172.21.15.69:6800/284433772 conn(0x7f2464038900 0x7f246403adb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:32.084 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.080+0000 7f247af28700 1 --2- 172.21.15.145:0/1769516935 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24741974e0 0x7f24741998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:32.084 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.080+0000 7f247af28700 1 -- 172.21.15.145:0/1769516935 >> 172.21.15.145:0/1769516935 conn(0x7f24740f9910 msgr2=0x7f24740fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:32.085 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.080+0000 7f247af28700 1 -- 172.21.15.145:0/1769516935 shutdown_connections 2023-07-12T15:24:32.085 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:24:32.080+0000 7f247af28700 1 -- 172.21.15.145:0/1769516935 wait complete. 2023-07-12T15:24:32.698 DEBUG:teuthology.orchestra.run.smithi145:mgr.x> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mgr.x.service 2023-07-12T15:24:32.700 INFO:tasks.cephadm:Deploying OSDs... 2023-07-12T15:24:32.701 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:24:32.701 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2023-07-12T15:24:32.719 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-12T15:24:32.720 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2023-07-12T15:24:32.729 INFO:journalctl@ceph.mgr.x.smithi145.stdout:-- Logs begin at Wed 2023-07-12 15:04:57 UTC. -- 2023-07-12T15:24:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-12T15:24:32.777 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-12T15:24:32.777 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 377499 Links: 1 2023-07-12T15:24:32.777 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-12T15:24:32.777 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2023-07-12T15:24:32.777 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-07-12 15:24:20.499003032 +0000 2023-07-12T15:24:32.778 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-07-12 15:14:24.583875905 +0000 2023-07-12T15:24:32.778 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-07-12 15:14:24.583875905 +0000 2023-07-12T15:24:32.778 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-07-12T15:24:32.778 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-12T15:24:32.842 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:32 smithi069 ceph-mon[94241]: Reconfiguring mon.b (monmap changed)... 2023-07-12T15:24:32.842 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:32 smithi069 ceph-mon[94241]: Reconfiguring daemon mon.b on smithi145 2023-07-12T15:24:32.842 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:32 smithi069 ceph-mon[94241]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:32.842 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:32 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:32.843 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:32 smithi069 ceph-mon[102777]: Reconfiguring mon.b (monmap changed)... 2023-07-12T15:24:32.843 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:32 smithi069 ceph-mon[102777]: Reconfiguring daemon mon.b on smithi145 2023-07-12T15:24:32.843 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:32 smithi069 ceph-mon[102777]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:32.843 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:32 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:32.844 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-07-12T15:24:32.844 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-07-12T15:24:32.844 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000248593 s, 2.1 MB/s 2023-07-12T15:24:32.845 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-12T15:24:32.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:32 smithi145 ceph-mon[95801]: Reconfiguring mon.b (monmap changed)... 2023-07-12T15:24:32.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:32 smithi145 ceph-mon[95801]: Reconfiguring daemon mon.b on smithi145 2023-07-12T15:24:32.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:32 smithi145 ceph-mon[95801]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:32.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:32 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:32.903 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2023-07-12T15:24:32.966 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-12T15:24:32.966 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-12T15:24:32.967 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 383177 Links: 1 2023-07-12T15:24:32.967 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-12T15:24:32.967 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2023-07-12T15:24:32.968 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-07-12 15:24:20.499003032 +0000 2023-07-12T15:24:32.968 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-07-12 15:14:25.150873420 +0000 2023-07-12T15:24:32.968 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-07-12 15:14:25.150873420 +0000 2023-07-12T15:24:32.969 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-07-12T15:24:32.969 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-12T15:24:33.031 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-07-12T15:24:33.031 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-07-12T15:24:33.032 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.00022327 s, 2.3 MB/s 2023-07-12T15:24:33.033 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-12T15:24:33.090 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2023-07-12T15:24:33.148 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-12T15:24:33.148 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-12T15:24:33.149 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 379482 Links: 1 2023-07-12T15:24:33.149 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-12T15:24:33.149 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2023-07-12T15:24:33.149 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-07-12 15:24:20.499003032 +0000 2023-07-12T15:24:33.149 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-07-12 15:14:25.653871215 +0000 2023-07-12T15:24:33.149 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-07-12 15:14:25.653871215 +0000 2023-07-12T15:24:33.150 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-07-12T15:24:33.150 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-12T15:24:33.213 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-07-12T15:24:33.213 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-07-12T15:24:33.213 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.00023244 s, 2.2 MB/s 2023-07-12T15:24:33.214 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-12T15:24:33.272 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2023-07-12T15:24:33.329 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-12T15:24:33.329 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-12T15:24:33.330 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 381748 Links: 1 2023-07-12T15:24:33.330 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-12T15:24:33.330 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2023-07-12T15:24:33.330 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-07-12 15:24:20.499003032 +0000 2023-07-12T15:24:33.330 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-07-12 15:14:26.157869006 +0000 2023-07-12T15:24:33.330 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-07-12 15:14:26.157869006 +0000 2023-07-12T15:24:33.330 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-07-12T15:24:33.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-12T15:24:33.394 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-07-12T15:24:33.396 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-07-12T15:24:33.396 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000212095 s, 2.4 MB/s 2023-07-12T15:24:33.398 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-12T15:24:33.458 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-07-12T15:24:33.458 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/scratch_devs of=/dev/stdout 2023-07-12T15:24:33.475 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-12T15:24:33.475 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_1 2023-07-12T15:24:33.533 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-12T15:24:33.534 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-12T15:24:33.534 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 6h/6d Inode: 372683 Links: 1 2023-07-12T15:24:33.534 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-12T15:24:33.534 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2023-07-12T15:24:33.534 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2023-07-12 15:24:12.285792893 +0000 2023-07-12T15:24:33.534 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2023-07-12 15:14:24.563940678 +0000 2023-07-12T15:24:33.534 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2023-07-12 15:14:24.563940678 +0000 2023-07-12T15:24:33.535 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2023-07-12T15:24:33.535 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-12T15:24:33.603 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2023-07-12T15:24:33.604 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2023-07-12T15:24:33.604 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.00177733 s, 288 kB/s 2023-07-12T15:24:33.605 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-12T15:24:33.663 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_2 2023-07-12T15:24:33.721 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-12T15:24:33.722 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-12T15:24:33.722 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 6h/6d Inode: 379344 Links: 1 2023-07-12T15:24:33.722 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-12T15:24:33.722 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2023-07-12T15:24:33.722 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2023-07-12 15:24:12.285792893 +0000 2023-07-12T15:24:33.723 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2023-07-12 15:14:25.101929050 +0000 2023-07-12T15:24:33.723 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2023-07-12 15:14:25.101929050 +0000 2023-07-12T15:24:33.723 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2023-07-12T15:24:33.723 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-12T15:24:33.788 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2023-07-12T15:24:33.789 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2023-07-12T15:24:33.789 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000221705 s, 2.3 MB/s 2023-07-12T15:24:33.791 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-12T15:24:33.850 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_3 2023-07-12T15:24:33.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:33 smithi145 ceph-mon[95801]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi069=y;smithi145=x", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:24:33.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:33 smithi145 ceph-mon[95801]: Saving service mgr spec with placement smithi069=y;smithi145=x;count:2 2023-07-12T15:24:33.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:33 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:33.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:33 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:33.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:33 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:33.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:33 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:33.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:33 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:33.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:33 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:33.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:33 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-12T15:24:33.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:33 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-12T15:24:33.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:33 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-12T15:24:33.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:33 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:33.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:33 smithi145 ceph-mon[95801]: Deploying daemon mgr.x on smithi145 2023-07-12T15:24:33.870 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-12T15:24:33.870 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-12T15:24:33.870 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 6h/6d Inode: 379496 Links: 1 2023-07-12T15:24:33.871 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-12T15:24:33.871 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2023-07-12T15:24:33.871 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2023-07-12 15:24:12.285792893 +0000 2023-07-12T15:24:33.872 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2023-07-12 15:14:25.600918297 +0000 2023-07-12T15:24:33.872 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2023-07-12 15:14:25.600918297 +0000 2023-07-12T15:24:33.872 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2023-07-12T15:24:33.873 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-12T15:24:33.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[94241]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi069=y;smithi145=x", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:24:33.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[94241]: Saving service mgr spec with placement smithi069=y;smithi145=x;count:2 2023-07-12T15:24:33.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:33.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:33.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:33.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:33.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:33.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:33.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-12T15:24:33.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-12T15:24:33.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-12T15:24:33.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:33.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[94241]: Deploying daemon mgr.x on smithi145 2023-07-12T15:24:33.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[102777]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi069=y;smithi145=x", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:24:33.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[102777]: Saving service mgr spec with placement smithi069=y;smithi145=x;count:2 2023-07-12T15:24:33.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:33.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:33.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:33.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:33.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:33.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:33.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-12T15:24:33.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-12T15:24:33.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-12T15:24:33.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:33.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:33 smithi069 ceph-mon[102777]: Deploying daemon mgr.x on smithi145 2023-07-12T15:24:34.005 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2023-07-12T15:24:34.005 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2023-07-12T15:24:34.005 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.00024783 s, 2.1 MB/s 2023-07-12T15:24:34.007 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-12T15:24:34.024 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_4 2023-07-12T15:24:34.081 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-12T15:24:34.081 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-12T15:24:34.081 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 6h/6d Inode: 374724 Links: 1 2023-07-12T15:24:34.081 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-12T15:24:34.082 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2023-07-12T15:24:34.082 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2023-07-12 15:24:33.956333636 +0000 2023-07-12T15:24:34.082 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2023-07-12 15:14:26.122907048 +0000 2023-07-12T15:24:34.082 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2023-07-12 15:14:26.122907048 +0000 2023-07-12T15:24:34.082 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2023-07-12T15:24:34.082 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-12T15:24:34.151 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2023-07-12T15:24:34.151 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2023-07-12T15:24:34.151 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000180871 s, 2.8 MB/s 2023-07-12T15:24:34.152 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-12T15:24:34.214 INFO:tasks.cephadm:Deploying osd.0 on smithi069 with /dev/vg_nvme/lv_4... 2023-07-12T15:24:34.216 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-12T15:24:34.350 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:34 smithi145 systemd[1]: Starting Ceph mgr.x for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:24:34.719 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:34 smithi145 ceph-mon[95801]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:34.720 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:34 smithi145 podman[97900]: 2023-07-12T15:24:34.720 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:34 smithi145 bash[97900]: d1c437ea422d03dda1a6f25ff77d47564bb5f90d876edfa8206a07d446b8b3fa 2023-07-12T15:24:34.720 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:34 smithi145 ceph-mgr[97940]: -- 172.21.15.145:0/3290355833 <== mon.1 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55804f7251e0 con 0x55804f6fb000 2023-07-12T15:24:34.775 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:24:34.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:34 smithi069 ceph-mon[94241]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:34.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:34 smithi069 ceph-mon[102777]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:35.011 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:34 smithi145 systemd[1]: Started Ceph mgr.x for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:24:35.012 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:34.831+0000 7faa3c382200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-12T15:24:35.013 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:34.915+0000 7faa3c382200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-12T15:24:35.287 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:35.009+0000 7faa3c382200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-12T15:24:35.288 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:35.110+0000 7faa3c382200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-12T15:24:35.288 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:35.285+0000 7faa3c382200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-12T15:24:35.941 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:35.608+0000 7faa3c382200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-12T15:24:35.942 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:35.703+0000 7faa3c382200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-12T15:24:35.942 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:35.840+0000 7faa3c382200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-12T15:24:35.942 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:35 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:35.942 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:35 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:35.943 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:35 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:35.943 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:35 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:35.943 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:35 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:36.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:35 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:35 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:35 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:35 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:35 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:36.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:35 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:35 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:35 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:35 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:35 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:36.304 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-12T15:24:36.304 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-12T15:24:36.304 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2023-07-12T15:24:36.304 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2023-07-12T15:24:36.305 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0149589 s, 701 MB/s 2023-07-12T15:24:36.305 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2023-07-12T15:24:36.350 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:35.939+0000 7faa3c382200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-12T15:24:36.727 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi069:vg_nvme/lv_4 2023-07-12T15:24:36.784 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:36 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:36.400+0000 7faa3c382200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-12T15:24:36.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[94241]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:36.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:36.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:36.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-12T15:24:36.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-12T15:24:36.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:36.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[102777]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:36.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:36.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:36.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:36.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-12T15:24:36.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-12T15:24:36.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:36 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:37.100 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:36 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:36.973+0000 7faa3c382200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-12T15:24:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:36 smithi145 ceph-mon[95801]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:36 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:36 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:36 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:37.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:36 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:37.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:36 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:37.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:36 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-12T15:24:37.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:36 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-12T15:24:37.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:36 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:37.600 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:37 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:37.144+0000 7faa3c382200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-12T15:24:37.670 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:24:37.946 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[102777]: Reconfiguring mgr.y (unknown last config time)... 2023-07-12T15:24:37.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[102777]: Reconfiguring daemon mgr.y on smithi069 2023-07-12T15:24:37.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:37.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:37.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:37.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:37.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:37.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:37.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[94241]: Reconfiguring mgr.y (unknown last config time)... 2023-07-12T15:24:37.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[94241]: Reconfiguring daemon mgr.y on smithi069 2023-07-12T15:24:37.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:37.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:37.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:37.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:37.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:37.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:38.100 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:37 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:37.902+0000 7faa3c382200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-12T15:24:38.101 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:37 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:37.996+0000 7faa3c382200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-12T15:24:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:37 smithi145 ceph-mon[95801]: Reconfiguring mgr.y (unknown last config time)... 2023-07-12T15:24:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:37 smithi145 ceph-mon[95801]: Reconfiguring daemon mgr.y on smithi069 2023-07-12T15:24:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:38.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:38.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:38.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:38.449 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:38 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:38.173+0000 7faa3c382200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-12T15:24:38.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:38 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:38.447+0000 7faa3c382200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-12T15:24:38.795 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:38 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:38.683+0000 7faa3c382200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-12T15:24:38.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.888+0000 7f533f546700 1 -- 172.21.15.69:0/1746641181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53380ff070 msgr2=0x7f53380ff480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:38.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.888+0000 7f533f546700 1 --2- 172.21.15.69:0/1746641181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53380ff070 0x7f53380ff480 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f53340098f0 tx=0x7f5334009c00 comp rx=0 tx=0).stop 2023-07-12T15:24:38.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.888+0000 7f533f546700 1 -- 172.21.15.69:0/1746641181 shutdown_connections 2023-07-12T15:24:38.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.888+0000 7f533f546700 1 --2- 172.21.15.69:0/1746641181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5338100c00 0x7f53381074c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:38.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.888+0000 7f533f546700 1 --2- 172.21.15.69:0/1746641181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53380ff070 0x7f53380ff480 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:38.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.888+0000 7f533f546700 1 --2- 172.21.15.69:0/1746641181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5338100270 0x7f53381006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:38.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.888+0000 7f533f546700 1 -- 172.21.15.69:0/1746641181 >> 172.21.15.69:0/1746641181 conn(0x7f53380fa6a0 msgr2=0x7f53380fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:38.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.888+0000 7f533f546700 1 -- 172.21.15.69:0/1746641181 shutdown_connections 2023-07-12T15:24:38.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.888+0000 7f533f546700 1 -- 172.21.15.69:0/1746641181 wait complete. 2023-07-12T15:24:38.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.888+0000 7f533f546700 1 Processor -- start 2023-07-12T15:24:38.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.888+0000 7f533f546700 1 -- start start 2023-07-12T15:24:38.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533f546700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53380ff070 0x7f53381977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:38.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533f546700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5338100270 0x7f5338197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:38.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533f546700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5338100c00 0x7f533819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:38.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533f546700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533819f370 con 0x7f5338100c00 2023-07-12T15:24:38.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533f546700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533819f4b0 con 0x7f5338100270 2023-07-12T15:24:38.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533f546700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533819f640 con 0x7f53380ff070 2023-07-12T15:24:38.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533d2e2700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53380ff070 0x7f53381977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:38.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533d2e2700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53380ff070 0x7f53381977e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:33370/0 (socket says 172.21.15.69:33370) 2023-07-12T15:24:38.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533dae3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5338100c00 0x7f533819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:38.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533d2e2700 1 -- 172.21.15.69:0/2280203258 learned_addr learned my addr 172.21.15.69:0/2280203258 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:24:38.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533cae1700 1 --2- 172.21.15.69:0/2280203258 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5338100270 0x7f5338197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:38.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533d2e2700 1 -- 172.21.15.69:0/2280203258 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5338100270 msgr2=0x7f5338197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:38.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533d2e2700 1 --2- 172.21.15.69:0/2280203258 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5338100270 0x7f5338197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:38.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533d2e2700 1 -- 172.21.15.69:0/2280203258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5338100c00 msgr2=0x7f533819eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:38.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533d2e2700 1 --2- 172.21.15.69:0/2280203258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5338100c00 0x7f533819eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:38.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533d2e2700 1 -- 172.21.15.69:0/2280203258 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5330009d00 con 0x7f53380ff070 2023-07-12T15:24:38.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533cae1700 1 --2- 172.21.15.69:0/2280203258 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5338100270 0x7f5338197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:24:38.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533dae3700 1 --2- 172.21.15.69:0/2280203258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5338100c00 0x7f533819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:24:38.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533d2e2700 1 --2- 172.21.15.69:0/2280203258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53380ff070 0x7f53381977e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f533400b790 tx=0x7f533400b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:38.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f532e7fc700 1 -- 172.21.15.69:0/2280203258 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533401c070 con 0x7f53380ff070 2023-07-12T15:24:38.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f532e7fc700 1 -- 172.21.15.69:0/2280203258 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f533400f7d0 con 0x7f53380ff070 2023-07-12T15:24:38.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f532e7fc700 1 -- 172.21.15.69:0/2280203258 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5334003cb0 con 0x7f53380ff070 2023-07-12T15:24:38.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533f546700 1 -- 172.21.15.69:0/2280203258 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5334009520 con 0x7f53380ff070 2023-07-12T15:24:38.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.889+0000 7f533f546700 1 -- 172.21.15.69:0/2280203258 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f533819fb40 con 0x7f53380ff070 2023-07-12T15:24:38.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.890+0000 7f532e7fc700 1 -- 172.21.15.69:0/2280203258 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 13) v1 ==== 45086+0+0 (secure 0 0 0) 0x7f533402a440 con 0x7f53380ff070 2023-07-12T15:24:38.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.890+0000 7f533f546700 1 -- 172.21.15.69:0/2280203258 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5338108ef0 con 0x7f53380ff070 2023-07-12T15:24:38.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.890+0000 7f532e7fc700 1 --2- 172.21.15.69:0/2280203258 >> v2:172.21.15.69:6800/284433772 conn(0x7f5324038640 0x7f532403aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:38.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.890+0000 7f532e7fc700 1 -- 172.21.15.69:0/2280203258 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f533404bbd0 con 0x7f53380ff070 2023-07-12T15:24:38.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.890+0000 7f533cae1700 1 --2- 172.21.15.69:0/2280203258 >> v2:172.21.15.69:6800/284433772 conn(0x7f5324038640 0x7f532403aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:38.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.891+0000 7f533cae1700 1 --2- 172.21.15.69:0/2280203258 >> v2:172.21.15.69:6800/284433772 conn(0x7f5324038640 0x7f532403aaf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5328005950 tx=0x7f53280058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:38.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:38.895+0000 7f532e7fc700 1 -- 172.21.15.69:0/2280203258 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f533401f760 con 0x7f53380ff070 2023-07-12T15:24:38.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:38 smithi069 ceph-mon[94241]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:38.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:38 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:38.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:38 smithi069 ceph-mon[102777]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:38.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:38 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:39.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:39.092+0000 7f533f546700 1 -- 172.21.15.69:0/2280203258 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5338106ca0 con 0x7f5324038640 2023-07-12T15:24:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:38 smithi145 ceph-mon[95801]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:38 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:39.101 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:38 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:38.846+0000 7faa3c382200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-12T15:24:39.600 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:39 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:39.138+0000 7faa3c382200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-12T15:24:40.100 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:39 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:39.899+0000 7faa3c382200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-12T15:24:40.101 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:40 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:40.036+0000 7faa3c382200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-12T15:24:40.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:39 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:24:40.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:39 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:24:40.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:39 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:40.110 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:39 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:24:40.110 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:39 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:24:40.110 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:39 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:40.110 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:39 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:24:40.111 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:39 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:24:40.111 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:39 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:40.600 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:40 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:40.233+0000 7faa3c382200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-12T15:24:40.601 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:40 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:40.379+0000 7faa3c382200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-12T15:24:41.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:40 smithi145 ceph-mon[95801]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:24:41.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:40 smithi145 ceph-mon[95801]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:41.101 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:40 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:40.612+0000 7faa3c382200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-12T15:24:41.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:40 smithi069 ceph-mon[94241]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:24:41.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:40 smithi069 ceph-mon[94241]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:41.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:40 smithi069 ceph-mon[102777]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:24:41.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:40 smithi069 ceph-mon[102777]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:41.801 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:24:41 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:24:41.500+0000 7faa3c382200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-12T15:24:41.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:41.806+0000 7f532e7fc700 1 -- 172.21.15.69:0/2280203258 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mgrmap(e 14) v1 ==== 89728+0+0 (secure 0 0 0) 0x7f533402a6f0 con 0x7f53380ff070 2023-07-12T15:24:42.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:41 smithi145 ceph-mon[95801]: Standby manager daemon x started 2023-07-12T15:24:42.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:41 smithi145 ceph-mon[95801]: from='mgr.? 172.21.15.145:0/564805546' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-12T15:24:42.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:41 smithi145 ceph-mon[95801]: from='mgr.? 172.21.15.145:0/564805546' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-12T15:24:42.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:41 smithi145 ceph-mon[95801]: from='mgr.? 172.21.15.145:0/564805546' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-12T15:24:42.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:41 smithi145 ceph-mon[95801]: from='mgr.? 172.21.15.145:0/564805546' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-12T15:24:42.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3989092195' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca84d141-6fa9-46e6-9fa4-2b6ebe1d0cad"}]: dispatch 2023-07-12T15:24:42.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:41 smithi145 ceph-mon[95801]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca84d141-6fa9-46e6-9fa4-2b6ebe1d0cad"}]: dispatch 2023-07-12T15:24:42.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:41 smithi145 ceph-mon[95801]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca84d141-6fa9-46e6-9fa4-2b6ebe1d0cad"}]': finished 2023-07-12T15:24:42.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:41 smithi145 ceph-mon[95801]: osdmap e5: 1 total, 0 up, 1 in 2023-07-12T15:24:42.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:41 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:42.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[94241]: Standby manager daemon x started 2023-07-12T15:24:42.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[94241]: from='mgr.? 172.21.15.145:0/564805546' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-12T15:24:42.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[94241]: from='mgr.? 172.21.15.145:0/564805546' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-12T15:24:42.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[94241]: from='mgr.? 172.21.15.145:0/564805546' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-12T15:24:42.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[94241]: from='mgr.? 172.21.15.145:0/564805546' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-12T15:24:42.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3989092195' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca84d141-6fa9-46e6-9fa4-2b6ebe1d0cad"}]: dispatch 2023-07-12T15:24:42.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[94241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca84d141-6fa9-46e6-9fa4-2b6ebe1d0cad"}]: dispatch 2023-07-12T15:24:42.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[94241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca84d141-6fa9-46e6-9fa4-2b6ebe1d0cad"}]': finished 2023-07-12T15:24:42.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[94241]: osdmap e5: 1 total, 0 up, 1 in 2023-07-12T15:24:42.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:42.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[102777]: Standby manager daemon x started 2023-07-12T15:24:42.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[102777]: from='mgr.? 172.21.15.145:0/564805546' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-12T15:24:42.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[102777]: from='mgr.? 172.21.15.145:0/564805546' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-12T15:24:42.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[102777]: from='mgr.? 172.21.15.145:0/564805546' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-12T15:24:42.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[102777]: from='mgr.? 172.21.15.145:0/564805546' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-12T15:24:42.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3989092195' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca84d141-6fa9-46e6-9fa4-2b6ebe1d0cad"}]: dispatch 2023-07-12T15:24:42.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[102777]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca84d141-6fa9-46e6-9fa4-2b6ebe1d0cad"}]: dispatch 2023-07-12T15:24:42.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[102777]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca84d141-6fa9-46e6-9fa4-2b6ebe1d0cad"}]': finished 2023-07-12T15:24:42.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[102777]: osdmap e5: 1 total, 0 up, 1 in 2023-07-12T15:24:42.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:41 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:43.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:42 smithi145 ceph-mon[95801]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:42 smithi145 ceph-mon[95801]: mgrmap e14: y(active, since 58s), standbys: x 2023-07-12T15:24:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:42 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-12T15:24:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3932680254' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:24:43.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:42 smithi069 ceph-mon[94241]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:42 smithi069 ceph-mon[94241]: mgrmap e14: y(active, since 58s), standbys: x 2023-07-12T15:24:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:42 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-12T15:24:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3932680254' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:24:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:42 smithi069 ceph-mon[102777]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:42 smithi069 ceph-mon[102777]: mgrmap e14: y(active, since 58s), standbys: x 2023-07-12T15:24:43.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:42 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-12T15:24:43.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3932680254' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:24:45.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:44 smithi145 ceph-mon[95801]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:45.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:44 smithi069 ceph-mon[94241]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:44 smithi069 ceph-mon[102777]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:45.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:45 smithi069 ceph-mon[94241]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:46.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:45 smithi145 ceph-mon[95801]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:46.228 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:45 smithi069 ceph-mon[102777]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:47.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:47 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-12T15:24:47.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:47 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:47.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:47 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-12T15:24:47.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:47 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:47.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:47 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-12T15:24:47.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:47 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:48.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:48 smithi069 ceph-mon[94241]: Deploying daemon osd.0 on smithi069 2023-07-12T15:24:48.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:48 smithi069 ceph-mon[94241]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:48.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:48 smithi069 ceph-mon[102777]: Deploying daemon osd.0 on smithi069 2023-07-12T15:24:48.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:48 smithi069 ceph-mon[102777]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:48.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:48 smithi145 ceph-mon[95801]: Deploying daemon osd.0 on smithi069 2023-07-12T15:24:48.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:48 smithi145 ceph-mon[95801]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:50.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:50 smithi145 ceph-mon[95801]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:50.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:50 smithi069 ceph-mon[94241]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:50.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:50 smithi069 ceph-mon[102777]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:52.879 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:52 smithi069 ceph-mon[94241]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:52.879 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:52 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:52.879 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:52 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:52.879 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:52 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:52.880 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:52 smithi069 ceph-mon[102777]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:52.880 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:52 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:52.880 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:52 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:52.881 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:52 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:53.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:52 smithi145 ceph-mon[95801]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:52 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:52 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:52 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:54.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:54.490+0000 7f532e7fc700 1 -- 172.21.15.69:0/2280203258 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5338106ca0 con 0x7f5324038640 2023-07-12T15:24:54.493 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 0 on host 'smithi069' 2023-07-12T15:24:54.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:54.493+0000 7f533f546700 1 -- 172.21.15.69:0/2280203258 >> v2:172.21.15.69:6800/284433772 conn(0x7f5324038640 msgr2=0x7f532403aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:54.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:54.493+0000 7f533f546700 1 --2- 172.21.15.69:0/2280203258 >> v2:172.21.15.69:6800/284433772 conn(0x7f5324038640 0x7f532403aaf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5328005950 tx=0x7f53280058e0 comp rx=0 tx=0).stop 2023-07-12T15:24:54.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:54.493+0000 7f533f546700 1 -- 172.21.15.69:0/2280203258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53380ff070 msgr2=0x7f53381977e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:54.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:54.493+0000 7f533f546700 1 --2- 172.21.15.69:0/2280203258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53380ff070 0x7f53381977e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f533400b790 tx=0x7f533400b7c0 comp rx=0 tx=0).stop 2023-07-12T15:24:54.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:54.493+0000 7f533f546700 1 -- 172.21.15.69:0/2280203258 shutdown_connections 2023-07-12T15:24:54.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:54.493+0000 7f533f546700 1 --2- 172.21.15.69:0/2280203258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5338100c00 0x7f533819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:54.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:54.493+0000 7f533f546700 1 --2- 172.21.15.69:0/2280203258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53380ff070 0x7f53381977e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:54.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:54.493+0000 7f533f546700 1 --2- 172.21.15.69:0/2280203258 >> v2:172.21.15.69:6800/284433772 conn(0x7f5324038640 0x7f532403aaf0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:54.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:54.493+0000 7f533f546700 1 --2- 172.21.15.69:0/2280203258 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5338100270 0x7f5338197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:54.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:54.493+0000 7f533f546700 1 -- 172.21.15.69:0/2280203258 >> 172.21.15.69:0/2280203258 conn(0x7f53380fa6a0 msgr2=0x7f53380fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:54.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:54.493+0000 7f533f546700 1 -- 172.21.15.69:0/2280203258 shutdown_connections 2023-07-12T15:24:54.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:54.493+0000 7f533f546700 1 -- 172.21.15.69:0/2280203258 wait complete. 2023-07-12T15:24:54.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:54 smithi069 ceph-mon[94241]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:54.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:54.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:54.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:54.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:54.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:54 smithi069 ceph-mon[102777]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:54.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:54 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:54.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:54 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:54.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:54 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:54.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:54 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:54.949 DEBUG:teuthology.orchestra.run.smithi069:osd.0> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.0.service 2023-07-12T15:24:54.952 INFO:tasks.cephadm:Deploying osd.1 on smithi069 with /dev/vg_nvme/lv_3... 2023-07-12T15:24:54.953 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-12T15:24:54.981 INFO:journalctl@ceph.osd.0.smithi069.stdout:-- Logs begin at Wed 2023-07-12 15:05:12 UTC. -- 2023-07-12T15:24:55.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:54 smithi145 ceph-mon[95801]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:55.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:54 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:55.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:54 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:55.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:54 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:55.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:54 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:55.705 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:24:55.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:55 smithi069 ceph-mon[102777]: from='osd.0 v2:172.21.15.69:6801/657400327' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-12T15:24:55.903 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:55 smithi069 ceph-mon[102777]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-12T15:24:55.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:55 smithi069 ceph-mon[94241]: from='osd.0 v2:172.21.15.69:6801/657400327' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-12T15:24:55.904 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:55 smithi069 ceph-mon[94241]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-12T15:24:56.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:55 smithi145 ceph-mon[95801]: from='osd.0 v2:172.21.15.69:6801/657400327' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-12T15:24:56.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:55 smithi145 ceph-mon[95801]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-12T15:24:56.806 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-mon[94241]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:56.807 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-mon[94241]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-12T15:24:56.807 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-mon[94241]: osdmap e6: 1 total, 0 up, 1 in 2023-07-12T15:24:56.807 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:56.807 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-mon[94241]: from='osd.0 v2:172.21.15.69:6801/657400327' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:24:56.807 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-mon[94241]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:24:56.808 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-mon[102777]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:56.808 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-mon[102777]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-12T15:24:56.808 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-mon[102777]: osdmap e6: 1 total, 0 up, 1 in 2023-07-12T15:24:56.808 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:56.808 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-mon[102777]: from='osd.0 v2:172.21.15.69:6801/657400327' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:24:56.808 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-mon[102777]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:24:56.809 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-0[107243]: 2023-07-12T15:24:56.664+0000 7f5ebce25700 -1 osd.0 0 waiting for initial osdmap 2023-07-12T15:24:56.809 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:24:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-0[107243]: 2023-07-12T15:24:56.666+0000 7f5eb5c14700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-12T15:24:56.999 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:56 smithi145 ceph-mon[95801]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:56.999 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:56 smithi145 ceph-mon[95801]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-12T15:24:56.999 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:56 smithi145 ceph-mon[95801]: osdmap e6: 1 total, 0 up, 1 in 2023-07-12T15:24:56.999 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:56 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:57.000 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:56 smithi145 ceph-mon[95801]: from='osd.0 v2:172.21.15.69:6801/657400327' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:24:57.000 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:56 smithi145 ceph-mon[95801]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:24:57.528 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-12T15:24:57.528 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-12T15:24:57.529 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2023-07-12T15:24:57.529 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2023-07-12T15:24:57.529 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0119922 s, 874 MB/s 2023-07-12T15:24:57.529 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2023-07-12T15:24:57.676 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:57 smithi069 ceph-mon[94241]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2023-07-12T15:24:57.677 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:57 smithi069 ceph-mon[94241]: osdmap e7: 1 total, 0 up, 1 in 2023-07-12T15:24:57.677 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:57 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:57.678 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:57 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:57.926 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:57 smithi069 ceph-mon[102777]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2023-07-12T15:24:57.926 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:57 smithi069 ceph-mon[102777]: osdmap e7: 1 total, 0 up, 1 in 2023-07-12T15:24:57.926 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:57 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:57.927 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:57 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:58.051 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:57 smithi145 ceph-mon[95801]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2023-07-12T15:24:58.052 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:57 smithi145 ceph-mon[95801]: osdmap e7: 1 total, 0 up, 1 in 2023-07-12T15:24:58.052 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:57 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:58.052 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:57 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:58.059 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi069:vg_nvme/lv_3 2023-07-12T15:24:58.613 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:24:58.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: purged_snaps scrub starts 2023-07-12T15:24:58.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: purged_snaps scrub ok 2023-07-12T15:24:58.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:58.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: osd.0 v2:172.21.15.69:6801/657400327 boot 2023-07-12T15:24:58.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: osdmap e8: 1 total, 1 up, 1 in 2023-07-12T15:24:58.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:58.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:58.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:58.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:24:58.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:58.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:58.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:58.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:58.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:58.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:58.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:58.927 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:58.928 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: purged_snaps scrub starts 2023-07-12T15:24:58.928 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: purged_snaps scrub ok 2023-07-12T15:24:58.928 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:58.928 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: osd.0 v2:172.21.15.69:6801/657400327 boot 2023-07-12T15:24:58.928 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: osdmap e8: 1 total, 1 up, 1 in 2023-07-12T15:24:58.928 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:58.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:58.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:58.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:24:58.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:58.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:58.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:58.929 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:58.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:58.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:58.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:58.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:59.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: purged_snaps scrub starts 2023-07-12T15:24:59.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: purged_snaps scrub ok 2023-07-12T15:24:59.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-12T15:24:59.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: osd.0 v2:172.21.15.69:6801/657400327 boot 2023-07-12T15:24:59.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: osdmap e8: 1 total, 1 up, 1 in 2023-07-12T15:24:59.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:24:59.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:59.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:59.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:24:59.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:59.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:59.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:59.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:59.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:24:59.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:24:59.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:24:59.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:24:59.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:59 smithi069 ceph-mon[94241]: Detected new or changed devices on smithi069 2023-07-12T15:24:59.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:59 smithi069 ceph-mon[94241]: Adjusting osd_memory_target on smithi069 to 16144M 2023-07-12T15:24:59.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:24:59 smithi069 ceph-mon[94241]: osdmap e9: 1 total, 1 up, 1 in 2023-07-12T15:24:59.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.921+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/2807103852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5c0100630 msgr2=0x7fa5c0100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:59.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.921+0000 7fa5c7b3d700 1 --2- 172.21.15.69:0/2807103852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5c0100630 0x7fa5c0100a40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00098f0 tx=0x7fa5b0009c00 comp rx=0 tx=0).stop 2023-07-12T15:24:59.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.921+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/2807103852 shutdown_connections 2023-07-12T15:24:59.927 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.921+0000 7fa5c7b3d700 1 --2- 172.21.15.69:0/2807103852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5c00ffc40 0x7fa5c01074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:59.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.921+0000 7fa5c7b3d700 1 --2- 172.21.15.69:0/2807103852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5c0100630 0x7fa5c0100a40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:59.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.921+0000 7fa5c7b3d700 1 --2- 172.21.15.69:0/2807103852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5c00ff290 0x7fa5c00ff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:59.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.921+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/2807103852 >> 172.21.15.69:0/2807103852 conn(0x7fa5c00f9e50 msgr2=0x7fa5c00fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:24:59.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.921+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/2807103852 shutdown_connections 2023-07-12T15:24:59.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.921+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/2807103852 wait complete. 2023-07-12T15:24:59.928 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c7b3d700 1 Processor -- start 2023-07-12T15:24:59.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c7b3d700 1 -- start start 2023-07-12T15:24:59.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c7b3d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5c00ff290 0x7fa5c00fece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:59.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c7b3d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5c00ffc40 0x7fa5c019a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:59.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c7b3d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5c0100630 0x7fa5c019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:59.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c7b3d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c019d0f0 con 0x7fa5c00ff290 2023-07-12T15:24:59.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c7b3d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c019d260 con 0x7fa5c0100630 2023-07-12T15:24:59.929 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c7b3d700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c019d420 con 0x7fa5c00ffc40 2023-07-12T15:24:59.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c58d9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5c00ff290 0x7fa5c00fece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:59.930 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c58d9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5c00ff290 0x7fa5c00fece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40000/0 (socket says 172.21.15.69:40000) 2023-07-12T15:24:59.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c50d8700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5c00ffc40 0x7fa5c019a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:59.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c58d9700 1 -- 172.21.15.69:0/4229094623 learned_addr learned my addr 172.21.15.69:0/4229094623 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:24:59.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c60da700 1 --2- 172.21.15.69:0/4229094623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5c0100630 0x7fa5c019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:59.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c58d9700 1 -- 172.21.15.69:0/4229094623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5c00ffc40 msgr2=0x7fa5c019a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:59.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c58d9700 1 --2- 172.21.15.69:0/4229094623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5c00ffc40 0x7fa5c019a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:59.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c58d9700 1 -- 172.21.15.69:0/4229094623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5c0100630 msgr2=0x7fa5c019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:24:59.931 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c58d9700 1 --2- 172.21.15.69:0/4229094623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5c0100630 0x7fa5c019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:24:59.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c58d9700 1 -- 172.21.15.69:0/4229094623 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc0095f0 con 0x7fa5c00ff290 2023-07-12T15:24:59.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.922+0000 7fa5c58d9700 1 --2- 172.21.15.69:0/4229094623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5c00ff290 0x7fa5c00fece0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0009fd0 tx=0x7fa5b000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:59.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.923+0000 7fa5b6ffd700 1 -- 172.21.15.69:0/4229094623 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b001c070 con 0x7fa5c00ff290 2023-07-12T15:24:59.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.923+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/4229094623 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b0009520 con 0x7fa5c00ff290 2023-07-12T15:24:59.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.923+0000 7fa5b6ffd700 1 -- 172.21.15.69:0/4229094623 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fa5b000f7d0 con 0x7fa5c00ff290 2023-07-12T15:24:59.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.923+0000 7fa5b6ffd700 1 -- 172.21.15.69:0/4229094623 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b00038e0 con 0x7fa5c00ff290 2023-07-12T15:24:59.932 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.923+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/4229094623 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c019d980 con 0x7fa5c00ff290 2023-07-12T15:24:59.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.924+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/4229094623 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c004e800 con 0x7fa5c00ff290 2023-07-12T15:24:59.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.924+0000 7fa5b6ffd700 1 -- 172.21.15.69:0/4229094623 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 14) v1 ==== 89728+0+0 (secure 0 0 0) 0x7fa5b0020030 con 0x7fa5c00ff290 2023-07-12T15:24:59.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.924+0000 7fa5b6ffd700 1 --2- 172.21.15.69:0/4229094623 >> v2:172.21.15.69:6800/284433772 conn(0x7fa5ac06c690 0x7fa5ac06eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:24:59.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.924+0000 7fa5b6ffd700 1 -- 172.21.15.69:0/4229094623 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1541+0+0 (secure 0 0 0) 0x7fa5b008b800 con 0x7fa5c00ff290 2023-07-12T15:24:59.933 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.924+0000 7fa5c50d8700 1 --2- 172.21.15.69:0/4229094623 >> v2:172.21.15.69:6800/284433772 conn(0x7fa5ac06c690 0x7fa5ac06eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:24:59.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.925+0000 7fa5c50d8700 1 --2- 172.21.15.69:0/4229094623 >> v2:172.21.15.69:6800/284433772 conn(0x7fa5ac06c690 0x7fa5ac06eb40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00ff260 tx=0x7fa5bc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:24:59.934 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:24:59.929+0000 7fa5b6ffd700 1 -- 172.21.15.69:0/4229094623 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa5b0056f10 con 0x7fa5c00ff290 2023-07-12T15:24:59.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:59 smithi069 ceph-mon[102777]: Detected new or changed devices on smithi069 2023-07-12T15:24:59.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:59 smithi069 ceph-mon[102777]: Adjusting osd_memory_target on smithi069 to 16144M 2023-07-12T15:24:59.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:24:59 smithi069 ceph-mon[102777]: osdmap e9: 1 total, 1 up, 1 in 2023-07-12T15:25:00.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:59 smithi145 ceph-mon[95801]: Detected new or changed devices on smithi069 2023-07-12T15:25:00.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:59 smithi145 ceph-mon[95801]: Adjusting osd_memory_target on smithi069 to 16144M 2023-07-12T15:25:00.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:24:59 smithi145 ceph-mon[95801]: osdmap e9: 1 total, 1 up, 1 in 2023-07-12T15:25:00.126 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:00.123+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/4229094623 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5c019dd20 con 0x7fa5ac06c690 2023-07-12T15:25:00.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:00 smithi069 ceph-mon[94241]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:00.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:00 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:25:00.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:00 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:25:00.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:00 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:00.950 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:00 smithi069 ceph-mon[102777]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:00.950 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:00 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:25:00.951 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:00 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:25:00.951 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:00 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:01.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:00 smithi145 ceph-mon[95801]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:01.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:00 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:25:01.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:00 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:25:01.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:00 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:01.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:01 smithi145 ceph-mon[95801]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:25:01.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:01 smithi069 ceph-mon[94241]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:25:01.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:01 smithi069 ceph-mon[102777]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:25:02.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:02 smithi069 ceph-mon[94241]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3228423702' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e1e1865-07a1-4490-af3b-48f832fe203e"}]: dispatch 2023-07-12T15:25:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3228423702' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e1e1865-07a1-4490-af3b-48f832fe203e"}]': finished 2023-07-12T15:25:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:02 smithi069 ceph-mon[94241]: osdmap e10: 2 total, 1 up, 2 in 2023-07-12T15:25:02.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:02.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:02 smithi069 ceph-mon[102777]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:02.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3228423702' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e1e1865-07a1-4490-af3b-48f832fe203e"}]: dispatch 2023-07-12T15:25:02.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3228423702' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e1e1865-07a1-4490-af3b-48f832fe203e"}]': finished 2023-07-12T15:25:02.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:02 smithi069 ceph-mon[102777]: osdmap e10: 2 total, 1 up, 2 in 2023-07-12T15:25:02.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:02 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:03.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:02 smithi145 ceph-mon[95801]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3228423702' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e1e1865-07a1-4490-af3b-48f832fe203e"}]: dispatch 2023-07-12T15:25:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3228423702' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e1e1865-07a1-4490-af3b-48f832fe203e"}]': finished 2023-07-12T15:25:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:02 smithi145 ceph-mon[95801]: osdmap e10: 2 total, 1 up, 2 in 2023-07-12T15:25:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:02 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:03.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1954380327' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:25:03.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1954380327' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:25:04.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1954380327' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:25:04.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:04 smithi069 ceph-mon[94241]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:04.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:04 smithi069 ceph-mon[102777]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:05.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:04 smithi145 ceph-mon[95801]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:06.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:05 smithi069 ceph-mon[94241]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:06.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:05 smithi069 ceph-mon[102777]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:06.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:05 smithi145 ceph-mon[95801]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:08.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:08 smithi069 ceph-mon[102777]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:08.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:08 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-12T15:25:08.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:08 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:08.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:08 smithi069 ceph-mon[94241]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:08.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:08 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-12T15:25:08.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:08 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:08.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:08 smithi145 ceph-mon[95801]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:08.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:08 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-12T15:25:08.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:08 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:09.773 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:09 smithi069 ceph-mon[94241]: Deploying daemon osd.1 on smithi069 2023-07-12T15:25:09.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:09 smithi069 ceph-mon[102777]: Deploying daemon osd.1 on smithi069 2023-07-12T15:25:09.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:09 smithi145 ceph-mon[95801]: Deploying daemon osd.1 on smithi069 2023-07-12T15:25:10.702 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:10 smithi069 ceph-mon[102777]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:10.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:10 smithi069 ceph-mon[94241]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:10.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:10 smithi145 ceph-mon[95801]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:12.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:12 smithi069 ceph-mon[94241]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:12.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:12 smithi069 ceph-mon[102777]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:12.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:12 smithi145 ceph-mon[95801]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:13.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:13 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:13.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:13 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:13.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:13 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:13.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:13 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:13.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:13 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:13.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:13 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:13.838 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:13 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:13.839 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:13 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:13.839 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:13 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:14.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:14 smithi145 ceph-mon[95801]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:14.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:14 smithi069 ceph-mon[94241]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:14.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:14 smithi069 ceph-mon[102777]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:15.476+0000 7fa5b6ffd700 1 -- 172.21.15.69:0/4229094623 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa5c019dd20 con 0x7fa5ac06c690 2023-07-12T15:25:15.479 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 1 on host 'smithi069' 2023-07-12T15:25:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:15.478+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/4229094623 >> v2:172.21.15.69:6800/284433772 conn(0x7fa5ac06c690 msgr2=0x7fa5ac06eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:25:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:15.478+0000 7fa5c7b3d700 1 --2- 172.21.15.69:0/4229094623 >> v2:172.21.15.69:6800/284433772 conn(0x7fa5ac06c690 0x7fa5ac06eb40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00ff260 tx=0x7fa5bc009310 comp rx=0 tx=0).stop 2023-07-12T15:25:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:15.478+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/4229094623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5c00ff290 msgr2=0x7fa5c00fece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:25:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:15.478+0000 7fa5c7b3d700 1 --2- 172.21.15.69:0/4229094623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5c00ff290 0x7fa5c00fece0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0009fd0 tx=0x7fa5b000b790 comp rx=0 tx=0).stop 2023-07-12T15:25:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:15.478+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/4229094623 shutdown_connections 2023-07-12T15:25:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:15.478+0000 7fa5c7b3d700 1 --2- 172.21.15.69:0/4229094623 >> v2:172.21.15.69:6800/284433772 conn(0x7fa5ac06c690 0x7fa5ac06eb40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:15.478+0000 7fa5c7b3d700 1 --2- 172.21.15.69:0/4229094623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5c0100630 0x7fa5c019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:15.478+0000 7fa5c7b3d700 1 --2- 172.21.15.69:0/4229094623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5c00ff290 0x7fa5c00fece0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:15.478+0000 7fa5c7b3d700 1 --2- 172.21.15.69:0/4229094623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5c00ffc40 0x7fa5c019a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:15.478+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/4229094623 >> 172.21.15.69:0/4229094623 conn(0x7fa5c00f9e50 msgr2=0x7fa5c00fc140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:25:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:15.478+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/4229094623 shutdown_connections 2023-07-12T15:25:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:15.478+0000 7fa5c7b3d700 1 -- 172.21.15.69:0/4229094623 wait complete. 2023-07-12T15:25:15.900 DEBUG:teuthology.orchestra.run.smithi069:osd.1> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.1.service 2023-07-12T15:25:15.902 INFO:tasks.cephadm:Deploying osd.2 on smithi069 with /dev/vg_nvme/lv_2... 2023-07-12T15:25:15.903 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-12T15:25:15.934 INFO:journalctl@ceph.osd.1.smithi069.stdout:-- Logs begin at Wed 2023-07-12 15:05:12 UTC. -- 2023-07-12T15:25:16.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:16.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:25:16.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:16.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[94241]: from='osd.1 v2:172.21.15.69:6805/2434701037' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-12T15:25:16.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:16.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:25:16.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:16.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:15 smithi069 ceph-mon[102777]: from='osd.1 v2:172.21.15.69:6805/2434701037' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-12T15:25:16.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:15 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:15 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:15 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:15 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:25:16.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:15 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:15 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:15 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:16.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:15 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:16.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:15 smithi145 ceph-mon[95801]: from='osd.1 v2:172.21.15.69:6805/2434701037' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-12T15:25:16.561 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:25:16.900 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:16 smithi069 ceph-mon[102777]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:16.901 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:16 smithi069 ceph-mon[102777]: from='osd.1 v2:172.21.15.69:6805/2434701037' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-12T15:25:16.901 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:16 smithi069 ceph-mon[102777]: osdmap e11: 2 total, 1 up, 2 in 2023-07-12T15:25:16.901 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:16 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:16.901 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:16 smithi069 ceph-mon[102777]: from='osd.1 v2:172.21.15.69:6805/2434701037' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:25:16.901 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:16 smithi069 ceph-mon[94241]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:16.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:16 smithi069 ceph-mon[94241]: from='osd.1 v2:172.21.15.69:6805/2434701037' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-12T15:25:16.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:16 smithi069 ceph-mon[94241]: osdmap e11: 2 total, 1 up, 2 in 2023-07-12T15:25:16.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:16 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:16.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:16 smithi069 ceph-mon[94241]: from='osd.1 v2:172.21.15.69:6805/2434701037' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:25:17.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:16 smithi145 ceph-mon[95801]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:16 smithi145 ceph-mon[95801]: from='osd.1 v2:172.21.15.69:6805/2434701037' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-12T15:25:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:16 smithi145 ceph-mon[95801]: osdmap e11: 2 total, 1 up, 2 in 2023-07-12T15:25:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:16 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:16 smithi145 ceph-mon[95801]: from='osd.1 v2:172.21.15.69:6805/2434701037' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:25:17.735 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:25:17 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-1[111786]: 2023-07-12T15:25:17.446+0000 7fab59431700 -1 osd.1 0 waiting for initial osdmap 2023-07-12T15:25:17.736 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:25:17 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-1[111786]: 2023-07-12T15:25:17.449+0000 7fab5121e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-12T15:25:18.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[94241]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:18.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[94241]: from='osd.1 v2:172.21.15.69:6805/2434701037' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2023-07-12T15:25:18.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[94241]: osdmap e12: 2 total, 1 up, 2 in 2023-07-12T15:25:18.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:18.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:18.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:18.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:18.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:18.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:18.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:18.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:18.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:25:18.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:18.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[102777]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:18.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[102777]: from='osd.1 v2:172.21.15.69:6805/2434701037' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2023-07-12T15:25:18.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[102777]: osdmap e12: 2 total, 1 up, 2 in 2023-07-12T15:25:18.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:18.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:18.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:18.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:18.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:18.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:18.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:18.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:18.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:25:18.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:18.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:18 smithi145 ceph-mon[95801]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:18 smithi145 ceph-mon[95801]: from='osd.1 v2:172.21.15.69:6805/2434701037' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2023-07-12T15:25:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:18 smithi145 ceph-mon[95801]: osdmap e12: 2 total, 1 up, 2 in 2023-07-12T15:25:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:25:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:18.927 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-12T15:25:18.927 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-12T15:25:18.927 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2023-07-12T15:25:18.928 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2023-07-12T15:25:18.928 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109121 s, 961 MB/s 2023-07-12T15:25:18.928 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2023-07-12T15:25:19.322 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi069:vg_nvme/lv_2 2023-07-12T15:25:19.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[94241]: purged_snaps scrub starts 2023-07-12T15:25:19.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[94241]: purged_snaps scrub ok 2023-07-12T15:25:19.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[94241]: Detected new or changed devices on smithi069 2023-07-12T15:25:19.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[94241]: Adjusting osd_memory_target on smithi069 to 8072M 2023-07-12T15:25:19.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:19.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[94241]: osd.1 v2:172.21.15.69:6805/2434701037 boot 2023-07-12T15:25:19.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[94241]: osdmap e13: 2 total, 2 up, 2 in 2023-07-12T15:25:19.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:19.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[102777]: purged_snaps scrub starts 2023-07-12T15:25:19.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[102777]: purged_snaps scrub ok 2023-07-12T15:25:19.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[102777]: Detected new or changed devices on smithi069 2023-07-12T15:25:19.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[102777]: Adjusting osd_memory_target on smithi069 to 8072M 2023-07-12T15:25:19.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:19.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[102777]: osd.1 v2:172.21.15.69:6805/2434701037 boot 2023-07-12T15:25:19.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[102777]: osdmap e13: 2 total, 2 up, 2 in 2023-07-12T15:25:19.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:19 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:19.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:19 smithi145 ceph-mon[95801]: purged_snaps scrub starts 2023-07-12T15:25:19.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:19 smithi145 ceph-mon[95801]: purged_snaps scrub ok 2023-07-12T15:25:19.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:19 smithi145 ceph-mon[95801]: Detected new or changed devices on smithi069 2023-07-12T15:25:19.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:19 smithi145 ceph-mon[95801]: Adjusting osd_memory_target on smithi069 to 8072M 2023-07-12T15:25:19.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:19 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:19.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:19 smithi145 ceph-mon[95801]: osd.1 v2:172.21.15.69:6805/2434701037 boot 2023-07-12T15:25:19.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:19 smithi145 ceph-mon[95801]: osdmap e13: 2 total, 2 up, 2 in 2023-07-12T15:25:19.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:19 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:25:19.875 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:25:20.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:20 smithi069 ceph-mon[94241]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:20.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:20 smithi069 ceph-mon[94241]: osdmap e14: 2 total, 2 up, 2 in 2023-07-12T15:25:20.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:20 smithi069 ceph-mon[102777]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:20.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:20 smithi069 ceph-mon[102777]: osdmap e14: 2 total, 2 up, 2 in 2023-07-12T15:25:20.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:20 smithi145 ceph-mon[95801]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-12T15:25:20.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:20 smithi145 ceph-mon[95801]: osdmap e14: 2 total, 2 up, 2 in 2023-07-12T15:25:21.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.092+0000 7f9a8d797700 1 -- 172.21.15.69:0/3779968298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a881035d0 msgr2=0x7f9a88103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:25:21.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.093+0000 7f9a8d797700 1 --2- 172.21.15.69:0/3779968298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a881035d0 0x7f9a88103a20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009990 tx=0x7f9a7c009ca0 comp rx=0 tx=0).stop 2023-07-12T15:25:21.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.093+0000 7f9a8d797700 1 -- 172.21.15.69:0/3779968298 shutdown_connections 2023-07-12T15:25:21.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.093+0000 7f9a8d797700 1 --2- 172.21.15.69:0/3779968298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a88103f60 0x7f9a8810a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:21.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.093+0000 7f9a8d797700 1 --2- 172.21.15.69:0/3779968298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a881023d0 0x7f9a881027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:21.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.093+0000 7f9a8d797700 1 --2- 172.21.15.69:0/3779968298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a881035d0 0x7f9a88103a20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:21.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.093+0000 7f9a8d797700 1 -- 172.21.15.69:0/3779968298 >> 172.21.15.69:0/3779968298 conn(0x7f9a880fda20 msgr2=0x7f9a880ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:25:21.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.093+0000 7f9a8d797700 1 -- 172.21.15.69:0/3779968298 shutdown_connections 2023-07-12T15:25:21.097 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.093+0000 7f9a8d797700 1 -- 172.21.15.69:0/3779968298 wait complete. 2023-07-12T15:25:21.098 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.093+0000 7f9a8d797700 1 Processor -- start 2023-07-12T15:25:21.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a8d797700 1 -- start start 2023-07-12T15:25:21.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a8d797700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a881023d0 0x7f9a88197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:25:21.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a8d797700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a881035d0 0x7f9a88197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:25:21.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a8d797700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a88103f60 0x7f9a8819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:25:21.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a8d797700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8819f3a0 con 0x7f9a88103f60 2023-07-12T15:25:21.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a8d797700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8819f4e0 con 0x7f9a881035d0 2023-07-12T15:25:21.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a8d797700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8819f670 con 0x7f9a881023d0 2023-07-12T15:25:21.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a86ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a881023d0 0x7f9a88197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:25:21.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a877fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a88103f60 0x7f9a8819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:25:21.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a86ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a881023d0 0x7f9a88197810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:33502/0 (socket says 172.21.15.69:33502) 2023-07-12T15:25:21.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a86ffd700 1 -- 172.21.15.69:0/1304845896 learned_addr learned my addr 172.21.15.69:0/1304845896 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:25:21.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a867fc700 1 --2- 172.21.15.69:0/1304845896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a881035d0 0x7f9a88197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:25:21.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a86ffd700 1 -- 172.21.15.69:0/1304845896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a881035d0 msgr2=0x7f9a88197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:25:21.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a86ffd700 1 --2- 172.21.15.69:0/1304845896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a881035d0 0x7f9a88197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:21.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a86ffd700 1 -- 172.21.15.69:0/1304845896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a88103f60 msgr2=0x7f9a8819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:25:21.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a86ffd700 1 --2- 172.21.15.69:0/1304845896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a88103f60 0x7f9a8819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:21.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a86ffd700 1 -- 172.21.15.69:0/1304845896 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a78009520 con 0x7f9a881023d0 2023-07-12T15:25:21.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.094+0000 7f9a877fe700 1 --2- 172.21.15.69:0/1304845896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a88103f60 0x7f9a8819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:25:21.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.095+0000 7f9a86ffd700 1 --2- 172.21.15.69:0/1304845896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a881023d0 0x7f9a88197810 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800ea80 tx=0x7f9a7800ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:25:21.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.095+0000 7f9a6ffff700 1 -- 172.21.15.69:0/1304845896 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7800cbd0 con 0x7f9a881023d0 2023-07-12T15:25:21.101 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.095+0000 7f9a6ffff700 1 -- 172.21.15.69:0/1304845896 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f9a7800cd30 con 0x7f9a881023d0 2023-07-12T15:25:21.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.095+0000 7f9a6ffff700 1 -- 172.21.15.69:0/1304845896 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78018680 con 0x7f9a881023d0 2023-07-12T15:25:21.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.095+0000 7f9a8d797700 1 -- 172.21.15.69:0/1304845896 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7c0095f0 con 0x7f9a881023d0 2023-07-12T15:25:21.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.095+0000 7f9a8d797700 1 -- 172.21.15.69:0/1304845896 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8819fc70 con 0x7f9a881023d0 2023-07-12T15:25:21.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.096+0000 7f9a8d797700 1 -- 172.21.15.69:0/1304845896 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a88109a10 con 0x7f9a881023d0 2023-07-12T15:25:21.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.096+0000 7f9a6ffff700 1 -- 172.21.15.69:0/1304845896 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 14) v1 ==== 89728+0+0 (secure 0 0 0) 0x7f9a78010430 con 0x7f9a881023d0 2023-07-12T15:25:21.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.096+0000 7f9a6ffff700 1 --2- 172.21.15.69:0/1304845896 >> v2:172.21.15.69:6800/284433772 conn(0x7f9a7006c690 0x7f9a7006eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:25:21.102 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.097+0000 7f9a6ffff700 1 -- 172.21.15.69:0/1304845896 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 1833+0+0 (secure 0 0 0) 0x7f9a78014070 con 0x7f9a881023d0 2023-07-12T15:25:21.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.097+0000 7f9a867fc700 1 --2- 172.21.15.69:0/1304845896 >> v2:172.21.15.69:6800/284433772 conn(0x7f9a7006c690 0x7f9a7006eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:25:21.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.097+0000 7f9a867fc700 1 --2- 172.21.15.69:0/1304845896 >> v2:172.21.15.69:6800/284433772 conn(0x7f9a7006c690 0x7f9a7006eb40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00b2e0 tx=0x7f9a7c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:25:21.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.101+0000 7f9a6ffff700 1 -- 172.21.15.69:0/1304845896 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9a780562a0 con 0x7f9a881023d0 2023-07-12T15:25:21.299 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:21.296+0000 7f9a8d797700 1 -- 172.21.15.69:0/1304845896 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a88060f40 con 0x7f9a7006c690 2023-07-12T15:25:21.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:21 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:25:21.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:21 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:25:21.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:21 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:21.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:25:21.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:25:21.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:21.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:21 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:25:21.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:21 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:25:21.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:21 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:22.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:22 smithi069 ceph-mon[94241]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:25:22.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:22 smithi069 ceph-mon[94241]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:22.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:22 smithi069 ceph-mon[102777]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:25:22.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:22 smithi069 ceph-mon[102777]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:22.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:22 smithi145 ceph-mon[95801]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:25:22.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:22 smithi145 ceph-mon[95801]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:24.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:24 smithi069 ceph-mon[94241]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:24.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/786726205' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67ce3cb4-9ddd-477c-a490-dd919b0cf1fd"}]: dispatch 2023-07-12T15:25:24.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/786726205' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67ce3cb4-9ddd-477c-a490-dd919b0cf1fd"}]': finished 2023-07-12T15:25:24.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:24 smithi069 ceph-mon[94241]: osdmap e15: 3 total, 2 up, 3 in 2023-07-12T15:25:24.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:24 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:24.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4035299951' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:25:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:24 smithi069 ceph-mon[102777]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/786726205' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67ce3cb4-9ddd-477c-a490-dd919b0cf1fd"}]: dispatch 2023-07-12T15:25:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/786726205' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67ce3cb4-9ddd-477c-a490-dd919b0cf1fd"}]': finished 2023-07-12T15:25:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:24 smithi069 ceph-mon[102777]: osdmap e15: 3 total, 2 up, 3 in 2023-07-12T15:25:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:24 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:24.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4035299951' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:25:24.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:24 smithi145 ceph-mon[95801]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:24.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/786726205' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67ce3cb4-9ddd-477c-a490-dd919b0cf1fd"}]: dispatch 2023-07-12T15:25:24.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/786726205' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67ce3cb4-9ddd-477c-a490-dd919b0cf1fd"}]': finished 2023-07-12T15:25:24.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:24 smithi145 ceph-mon[95801]: osdmap e15: 3 total, 2 up, 3 in 2023-07-12T15:25:24.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:24 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:24.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4035299951' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:25:26.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:25 smithi069 ceph-mon[94241]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:26.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:25 smithi069 ceph-mon[102777]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:26.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:25 smithi145 ceph-mon[95801]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:28.674 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:28 smithi069 ceph-mon[94241]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:28.674 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:28 smithi069 ceph-mon[102777]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:28.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:28 smithi145 ceph-mon[95801]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:29.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:29 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-12T15:25:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:29 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:29 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-12T15:25:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:29 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:29.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:29 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-12T15:25:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:29 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:30.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:30 smithi069 ceph-mon[94241]: Deploying daemon osd.2 on smithi069 2023-07-12T15:25:30.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:30 smithi069 ceph-mon[94241]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:30.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:30 smithi069 ceph-mon[102777]: Deploying daemon osd.2 on smithi069 2023-07-12T15:25:30.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:30 smithi069 ceph-mon[102777]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:30.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:30 smithi145 ceph-mon[95801]: Deploying daemon osd.2 on smithi069 2023-07-12T15:25:30.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:30 smithi145 ceph-mon[95801]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:32.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:32 smithi069 ceph-mon[94241]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:32 smithi069 ceph-mon[102777]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:32.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:32 smithi145 ceph-mon[95801]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:34.880 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:34 smithi069 ceph-mon[102777]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:34.880 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:34 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:34.880 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:34 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:34.881 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:34 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:34.881 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:34 smithi069 ceph-mon[94241]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:34.881 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:34 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:34.881 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:34 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:34.881 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:34 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:35.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:34 smithi145 ceph-mon[95801]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:35.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:34 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:35.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:34 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:35.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:34 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:36.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:35 smithi069 ceph-mon[94241]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:36.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:35 smithi069 ceph-mon[102777]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:36.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:35 smithi145 ceph-mon[95801]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:36.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:36.493+0000 7f9a6ffff700 1 -- 172.21.15.69:0/1304845896 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9a88060f40 con 0x7f9a7006c690 2023-07-12T15:25:36.497 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 2 on host 'smithi069' 2023-07-12T15:25:36.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:36.496+0000 7f9a8d797700 1 -- 172.21.15.69:0/1304845896 >> v2:172.21.15.69:6800/284433772 conn(0x7f9a7006c690 msgr2=0x7f9a7006eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:25:36.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:36.496+0000 7f9a8d797700 1 --2- 172.21.15.69:0/1304845896 >> v2:172.21.15.69:6800/284433772 conn(0x7f9a7006c690 0x7f9a7006eb40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00b2e0 tx=0x7f9a7c005960 comp rx=0 tx=0).stop 2023-07-12T15:25:36.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:36.496+0000 7f9a8d797700 1 -- 172.21.15.69:0/1304845896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a881023d0 msgr2=0x7f9a88197810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:25:36.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:36.496+0000 7f9a8d797700 1 --2- 172.21.15.69:0/1304845896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a881023d0 0x7f9a88197810 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800ea80 tx=0x7f9a7800ed90 comp rx=0 tx=0).stop 2023-07-12T15:25:36.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:36.496+0000 7f9a8d797700 1 -- 172.21.15.69:0/1304845896 shutdown_connections 2023-07-12T15:25:36.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:36.496+0000 7f9a8d797700 1 --2- 172.21.15.69:0/1304845896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a88103f60 0x7f9a8819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:36.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:36.496+0000 7f9a8d797700 1 --2- 172.21.15.69:0/1304845896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a881023d0 0x7f9a88197810 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:36.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:36.496+0000 7f9a8d797700 1 --2- 172.21.15.69:0/1304845896 >> v2:172.21.15.69:6800/284433772 conn(0x7f9a7006c690 0x7f9a7006eb40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:36.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:36.496+0000 7f9a8d797700 1 --2- 172.21.15.69:0/1304845896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a881035d0 0x7f9a88197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:36.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:36.496+0000 7f9a8d797700 1 -- 172.21.15.69:0/1304845896 >> 172.21.15.69:0/1304845896 conn(0x7f9a880fda20 msgr2=0x7f9a88106b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:25:36.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:36.496+0000 7f9a8d797700 1 -- 172.21.15.69:0/1304845896 shutdown_connections 2023-07-12T15:25:36.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:36.496+0000 7f9a8d797700 1 -- 172.21.15.69:0/1304845896 wait complete. 2023-07-12T15:25:36.963 DEBUG:teuthology.orchestra.run.smithi069:osd.2> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.2.service 2023-07-12T15:25:36.965 INFO:tasks.cephadm:Deploying osd.3 on smithi069 with /dev/vg_nvme/lv_1... 2023-07-12T15:25:36.966 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-12T15:25:36.997 INFO:journalctl@ceph.osd.2.smithi069.stdout:-- Logs begin at Wed 2023-07-12 15:05:12 UTC. -- 2023-07-12T15:25:37.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:25:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:37.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:37 smithi145 ceph-mon[95801]: from='osd.2 v2:172.21.15.69:6809/1319832966' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-12T15:25:37.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:37 smithi145 ceph-mon[95801]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-12T15:25:37.658 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.658 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.658 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:37.658 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:25:37.658 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.659 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.659 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.659 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:37.659 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[102777]: from='osd.2 v2:172.21.15.69:6809/1319832966' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-12T15:25:37.659 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[102777]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-12T15:25:37.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.660 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.660 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:37.660 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:25:37.660 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.660 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.660 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:37.660 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:37.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[94241]: from='osd.2 v2:172.21.15.69:6809/1319832966' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-12T15:25:37.661 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:37 smithi069 ceph-mon[94241]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-12T15:25:37.726 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:25:38.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-mon[94241]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:38.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-mon[94241]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-12T15:25:38.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-mon[94241]: osdmap e16: 3 total, 2 up, 3 in 2023-07-12T15:25:38.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:38.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-mon[94241]: from='osd.2 v2:172.21.15.69:6809/1319832966' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:25:38.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-mon[94241]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:25:38.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-mon[102777]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:38.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-mon[102777]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-12T15:25:38.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-mon[102777]: osdmap e16: 3 total, 2 up, 3 in 2023-07-12T15:25:38.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:38.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-mon[102777]: from='osd.2 v2:172.21.15.69:6809/1319832966' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:25:38.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-mon[102777]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:25:38.487 INFO:journalctl@ceph.osd.2.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-2[116331]: 2023-07-12T15:25:38.477+0000 7f00d11fb700 -1 osd.2 0 waiting for initial osdmap 2023-07-12T15:25:38.487 INFO:journalctl@ceph.osd.2.smithi069.stdout:Jul 12 15:25:38 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-2[116331]: 2023-07-12T15:25:38.482+0000 7f00ca7eb700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-12T15:25:38.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:38 smithi145 ceph-mon[95801]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:38 smithi145 ceph-mon[95801]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-12T15:25:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:38 smithi145 ceph-mon[95801]: osdmap e16: 3 total, 2 up, 3 in 2023-07-12T15:25:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:38 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:38 smithi145 ceph-mon[95801]: from='osd.2 v2:172.21.15.69:6809/1319832966' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:25:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:38 smithi145 ceph-mon[95801]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:25:39.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:39 smithi069 ceph-mon[94241]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2023-07-12T15:25:39.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:39 smithi069 ceph-mon[94241]: osdmap e17: 3 total, 2 up, 3 in 2023-07-12T15:25:39.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:39 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:39.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:39 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:39.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:39 smithi069 ceph-mon[102777]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2023-07-12T15:25:39.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:39 smithi069 ceph-mon[102777]: osdmap e17: 3 total, 2 up, 3 in 2023-07-12T15:25:39.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:39 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:39.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:39 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:39.755 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-12T15:25:39.756 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-12T15:25:39.756 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2023-07-12T15:25:39.756 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2023-07-12T15:25:39.756 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0137758 s, 761 MB/s 2023-07-12T15:25:39.756 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2023-07-12T15:25:39.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:39 smithi145 ceph-mon[95801]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2023-07-12T15:25:39.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:39 smithi145 ceph-mon[95801]: osdmap e17: 3 total, 2 up, 3 in 2023-07-12T15:25:39.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:39 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:39.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:39 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:40.225 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi069:vg_nvme/lv_1 2023-07-12T15:25:40.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: purged_snaps scrub starts 2023-07-12T15:25:40.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: purged_snaps scrub ok 2023-07-12T15:25:40.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:40.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:40.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: osd.2 v2:172.21.15.69:6809/1319832966 boot 2023-07-12T15:25:40.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: osdmap e18: 3 total, 3 up, 3 in 2023-07-12T15:25:40.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:40.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:25:40.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: Detected new or changed devices on smithi069 2023-07-12T15:25:40.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:40.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:40.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:40.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:40.742 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:40.742 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: Adjusting osd_memory_target on smithi069 to 5381M 2023-07-12T15:25:40.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:40.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:40.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:25:40.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:40.744 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: purged_snaps scrub starts 2023-07-12T15:25:40.744 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: purged_snaps scrub ok 2023-07-12T15:25:40.744 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:40.745 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:40.745 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: osd.2 v2:172.21.15.69:6809/1319832966 boot 2023-07-12T15:25:40.745 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: osdmap e18: 3 total, 3 up, 3 in 2023-07-12T15:25:40.745 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:40.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:25:40.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: Detected new or changed devices on smithi069 2023-07-12T15:25:40.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:40.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:40.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:40.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:40.746 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:40.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: Adjusting osd_memory_target on smithi069 to 5381M 2023-07-12T15:25:40.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:40.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:40.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:25:40.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:40.776 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:25:40.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: purged_snaps scrub starts 2023-07-12T15:25:40.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: purged_snaps scrub ok 2023-07-12T15:25:40.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-12T15:25:40.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:40.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: osd.2 v2:172.21.15.69:6809/1319832966 boot 2023-07-12T15:25:40.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: osdmap e18: 3 total, 3 up, 3 in 2023-07-12T15:25:40.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:25:40.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:25:40.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: Detected new or changed devices on smithi069 2023-07-12T15:25:40.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:40.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:40.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:40.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:40.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:25:40.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: Adjusting osd_memory_target on smithi069 to 5381M 2023-07-12T15:25:40.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:40.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:40.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:25:40.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:41.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:41 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-12T15:25:41.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:41 smithi069 ceph-mon[94241]: osdmap e19: 3 total, 3 up, 3 in 2023-07-12T15:25:41.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:41 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:25:41.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:41 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-12T15:25:41.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:41 smithi069 ceph-mon[102777]: osdmap e19: 3 total, 3 up, 3 in 2023-07-12T15:25:41.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:41 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:25:41.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:41 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-12T15:25:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:41 smithi145 ceph-mon[95801]: osdmap e19: 3 total, 3 up, 3 in 2023-07-12T15:25:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:41 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:25:41.980 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.976+0000 7ff64dad3700 1 -- 172.21.15.69:0/406966370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6481013b0 msgr2=0x7ff648101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:25:41.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.976+0000 7ff64dad3700 1 --2- 172.21.15.69:0/406966370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6481013b0 0x7ff648101800 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff638009990 tx=0x7ff638009ca0 comp rx=0 tx=0).stop 2023-07-12T15:25:41.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.976+0000 7ff64dad3700 1 -- 172.21.15.69:0/406966370 shutdown_connections 2023-07-12T15:25:41.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.976+0000 7ff64dad3700 1 --2- 172.21.15.69:0/406966370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff648101d40 0x7ff648108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:41.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.976+0000 7ff64dad3700 1 --2- 172.21.15.69:0/406966370 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6481001b0 0x7ff6481005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:41.981 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.976+0000 7ff64dad3700 1 --2- 172.21.15.69:0/406966370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6481013b0 0x7ff648101800 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:41.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.976+0000 7ff64dad3700 1 -- 172.21.15.69:0/406966370 >> 172.21.15.69:0/406966370 conn(0x7ff6480fb820 msgr2=0x7ff6480fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:25:41.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.976+0000 7ff64dad3700 1 -- 172.21.15.69:0/406966370 shutdown_connections 2023-07-12T15:25:41.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.976+0000 7ff64dad3700 1 -- 172.21.15.69:0/406966370 wait complete. 2023-07-12T15:25:41.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff64dad3700 1 Processor -- start 2023-07-12T15:25:41.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff64dad3700 1 -- start start 2023-07-12T15:25:41.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff64dad3700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6481001b0 0x7ff648195500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:25:41.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff64dad3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6481013b0 0x7ff648195a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:25:41.982 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff64dad3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff648101d40 0x7ff64819cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:25:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff64dad3700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64819d090 con 0x7ff648101d40 2023-07-12T15:25:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff64dad3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64819d1d0 con 0x7ff6481013b0 2023-07-12T15:25:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff64dad3700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff64819d360 con 0x7ff6481001b0 2023-07-12T15:25:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff6477fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6481001b0 0x7ff648195500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:25:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff6477fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6481001b0 0x7ff648195500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:33588/0 (socket says 172.21.15.69:33588) 2023-07-12T15:25:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff6477fe700 1 -- 172.21.15.69:0/651812656 learned_addr learned my addr 172.21.15.69:0/651812656 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:25:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff647fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff648101d40 0x7ff64819cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:25:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff646ffd700 1 --2- 172.21.15.69:0/651812656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6481013b0 0x7ff648195a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:25:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff6477fe700 1 -- 172.21.15.69:0/651812656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6481013b0 msgr2=0x7ff648195a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:25:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff6477fe700 1 --2- 172.21.15.69:0/651812656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6481013b0 0x7ff648195a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff6477fe700 1 -- 172.21.15.69:0/651812656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff648101d40 msgr2=0x7ff64819cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:25:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff6477fe700 1 --2- 172.21.15.69:0/651812656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff648101d40 0x7ff64819cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff6477fe700 1 -- 172.21.15.69:0/651812656 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6380095f0 con 0x7ff6481001b0 2023-07-12T15:25:41.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.977+0000 7ff647fff700 1 --2- 172.21.15.69:0/651812656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff648101d40 0x7ff64819cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:25:41.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.978+0000 7ff6477fe700 1 --2- 172.21.15.69:0/651812656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6481001b0 0x7ff648195500 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00c9c0 tx=0x7ff63c00ccd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:25:41.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.978+0000 7ff644ff9700 1 -- 172.21.15.69:0/651812656 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c007ac0 con 0x7ff6481001b0 2023-07-12T15:25:41.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.978+0000 7ff644ff9700 1 -- 172.21.15.69:0/651812656 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7ff63c00ce60 con 0x7ff6481001b0 2023-07-12T15:25:41.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.978+0000 7ff644ff9700 1 -- 172.21.15.69:0/651812656 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c018400 con 0x7ff6481001b0 2023-07-12T15:25:41.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.978+0000 7ff64dad3700 1 -- 172.21.15.69:0/651812656 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64819d640 con 0x7ff6481001b0 2023-07-12T15:25:41.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.978+0000 7ff64dad3700 1 -- 172.21.15.69:0/651812656 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64819daf0 con 0x7ff6481001b0 2023-07-12T15:25:41.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.979+0000 7ff64dad3700 1 -- 172.21.15.69:0/651812656 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff648107760 con 0x7ff6481001b0 2023-07-12T15:25:41.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.979+0000 7ff644ff9700 1 -- 172.21.15.69:0/651812656 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 14) v1 ==== 89728+0+0 (secure 0 0 0) 0x7ff63c00f450 con 0x7ff6481001b0 2023-07-12T15:25:41.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.980+0000 7ff644ff9700 1 --2- 172.21.15.69:0/651812656 >> v2:172.21.15.69:6800/284433772 conn(0x7ff63006c690 0x7ff63006eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:25:41.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.980+0000 7ff644ff9700 1 -- 172.21.15.69:0/651812656 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2511+0+0 (secure 0 0 0) 0x7ff63c08a980 con 0x7ff6481001b0 2023-07-12T15:25:41.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.980+0000 7ff646ffd700 1 --2- 172.21.15.69:0/651812656 >> v2:172.21.15.69:6800/284433772 conn(0x7ff63006c690 0x7ff63006eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:25:41.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.980+0000 7ff646ffd700 1 --2- 172.21.15.69:0/651812656 >> v2:172.21.15.69:6800/284433772 conn(0x7ff63006c690 0x7ff63006eb40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff63800b2e0 tx=0x7ff638005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:25:41.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:41.984+0000 7ff644ff9700 1 -- 172.21.15.69:0/651812656 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff63c055cb0 con 0x7ff6481001b0 2023-07-12T15:25:42.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:42.203+0000 7ff64dad3700 1 -- 172.21.15.69:0/651812656 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff648060f40 con 0x7ff63006c690 2023-07-12T15:25:42.733 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:25:42 smithi069 sudo[118772]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-12T15:25:42.733 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:25:42 smithi069 sudo[118772]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-12T15:25:42.733 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:25:42 smithi069 sudo[118772]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-12T15:25:42.733 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:25:42 smithi069 sudo[118772]: pam_unix(sudo:session): session closed for user root 2023-07-12T15:25:42.734 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:42 smithi069 ceph-mon[94241]: pgmap v68: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:42.734 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:42 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-12T15:25:42.734 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:42 smithi069 ceph-mon[94241]: osdmap e20: 3 total, 3 up, 3 in 2023-07-12T15:25:42.734 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:42 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:25:42.734 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:42 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:25:42.734 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:42 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:42.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:42 smithi069 ceph-mon[102777]: pgmap v68: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:42.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:42 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-12T15:25:42.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:42 smithi069 ceph-mon[102777]: osdmap e20: 3 total, 3 up, 3 in 2023-07-12T15:25:42.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:42 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:25:42.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:42 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:25:42.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:42 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:42.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:42 smithi145 ceph-mon[95801]: pgmap v68: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:42.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:42 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-12T15:25:42.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:42 smithi145 ceph-mon[95801]: osdmap e20: 3 total, 3 up, 3 in 2023-07-12T15:25:42.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:42 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:25:42.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:42 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:25:42.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:42 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:42.985 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:25:42 smithi069 sudo[118805]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-12T15:25:42.985 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:25:42 smithi069 sudo[118805]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-12T15:25:42.986 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:25:42 smithi069 sudo[118805]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-12T15:25:42.986 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:25:42 smithi069 sudo[118805]: pam_unix(sudo:session): session closed for user root 2023-07-12T15:25:43.284 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:25:43 smithi069 sudo[118823]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-12T15:25:43.284 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:25:43 smithi069 sudo[118823]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-12T15:25:43.284 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:25:43 smithi069 sudo[118823]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-12T15:25:43.284 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:25:43 smithi069 sudo[118823]: pam_unix(sudo:session): session closed for user root 2023-07-12T15:25:43.538 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:43 smithi069 ceph-mon[102777]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:25:43.538 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:43 smithi069 ceph-mon[102777]: osdmap e21: 3 total, 3 up, 3 in 2023-07-12T15:25:43.538 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:43 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:43.539 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:43 smithi069 ceph-mon[94241]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:25:43.539 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:43 smithi069 ceph-mon[94241]: osdmap e21: 3 total, 3 up, 3 in 2023-07-12T15:25:43.539 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:43 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:43.539 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:25:43 smithi069 sudo[118865]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-12T15:25:43.540 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:25:43 smithi069 sudo[118865]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-12T15:25:43.540 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:25:43 smithi069 sudo[118865]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-12T15:25:43.540 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:25:43 smithi069 sudo[118865]: pam_unix(sudo:session): session closed for user root 2023-07-12T15:25:43.798 INFO:journalctl@ceph.osd.2.smithi069.stdout:Jul 12 15:25:43 smithi069 sudo[118875]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-12T15:25:43.798 INFO:journalctl@ceph.osd.2.smithi069.stdout:Jul 12 15:25:43 smithi069 sudo[118875]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-12T15:25:43.798 INFO:journalctl@ceph.osd.2.smithi069.stdout:Jul 12 15:25:43 smithi069 sudo[118875]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-12T15:25:43.798 INFO:journalctl@ceph.osd.2.smithi069.stdout:Jul 12 15:25:43 smithi069 sudo[118875]: pam_unix(sudo:session): session closed for user root 2023-07-12T15:25:43.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:43 smithi145 ceph-mon[95801]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:25:43.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:43 smithi145 ceph-mon[95801]: osdmap e21: 3 total, 3 up, 3 in 2023-07-12T15:25:43.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:43 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:44.331 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 sudo[118935]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-12T15:25:44.331 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 sudo[118935]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-12T15:25:44.332 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 sudo[118935]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-12T15:25:44.332 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 sudo[118935]: pam_unix(sudo:session): session closed for user root 2023-07-12T15:25:44.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 sudo[118969]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-12T15:25:44.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 sudo[118969]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-12T15:25:44.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 sudo[118969]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-12T15:25:44.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 sudo[118969]: pam_unix(sudo:session): session closed for user root 2023-07-12T15:25:44.827 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 sudo[98798]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-12T15:25:44.828 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 sudo[98798]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-12T15:25:44.828 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 sudo[98798]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-12T15:25:44.828 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 sudo[98798]: pam_unix(sudo:session): session closed for user root 2023-07-12T15:25:44.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:44.850+0000 7ff644ff9700 1 -- 172.21.15.69:0/651812656 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mgrmap(e 15) v1 ==== 89784+0+0 (secure 0 0 0) 0x7ff63c00f700 con 0x7ff6481001b0 2023-07-12T15:25:45.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 sudo[98801]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-12T15:25:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 sudo[98801]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-12T15:25:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 sudo[98801]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-12T15:25:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 sudo[98801]: pam_unix(sudo:session): session closed for user root 2023-07-12T15:25:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 ceph-mon[95801]: pgmap v71: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 ceph-mon[95801]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-12T15:25:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 ceph-mon[95801]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-12T15:25:45.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:25:45.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:25:45.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:25:45.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:25:45.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:25:45.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:25:45.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/175127167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a5e5d11-ad42-4663-8c6b-d04e9b968c07"}]: dispatch 2023-07-12T15:25:45.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:44 smithi145 ceph-mon[95801]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a5e5d11-ad42-4663-8c6b-d04e9b968c07"}]: dispatch 2023-07-12T15:25:45.107 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[102777]: pgmap v71: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:45.107 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[102777]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-12T15:25:45.108 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[102777]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-12T15:25:45.108 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:25:45.108 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:25:45.108 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:25:45.108 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:25:45.108 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:25:45.109 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:25:45.110 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/175127167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a5e5d11-ad42-4663-8c6b-d04e9b968c07"}]: dispatch 2023-07-12T15:25:45.110 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[102777]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a5e5d11-ad42-4663-8c6b-d04e9b968c07"}]: dispatch 2023-07-12T15:25:45.110 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[94241]: pgmap v71: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:45.111 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[94241]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-12T15:25:45.111 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[94241]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-12T15:25:45.111 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:25:45.111 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:25:45.111 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:25:45.112 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:25:45.112 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:25:45.112 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:25:45.112 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/175127167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a5e5d11-ad42-4663-8c6b-d04e9b968c07"}]: dispatch 2023-07-12T15:25:45.112 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:44 smithi069 ceph-mon[94241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a5e5d11-ad42-4663-8c6b-d04e9b968c07"}]: dispatch 2023-07-12T15:25:45.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 sudo[118980]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-12T15:25:45.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 sudo[118980]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-12T15:25:45.370 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 sudo[118980]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-12T15:25:45.371 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 sudo[118980]: pam_unix(sudo:session): session closed for user root 2023-07-12T15:25:45.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 sudo[119011]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-12T15:25:45.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 sudo[119011]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-12T15:25:45.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 sudo[119011]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-12T15:25:45.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 sudo[119011]: pam_unix(sudo:session): session closed for user root 2023-07-12T15:25:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:45 smithi145 ceph-mon[95801]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-12T15:25:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:45 smithi145 ceph-mon[95801]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-12T15:25:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:45 smithi145 ceph-mon[95801]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a5e5d11-ad42-4663-8c6b-d04e9b968c07"}]': finished 2023-07-12T15:25:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:45 smithi145 ceph-mon[95801]: osdmap e22: 4 total, 3 up, 4 in 2023-07-12T15:25:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:45 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:25:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:45 smithi145 ceph-mon[95801]: mgrmap e15: y(active, since 2m), standbys: x 2023-07-12T15:25:46.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:45 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:46.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:45 smithi145 ceph-mon[95801]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-12T15:25:46.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:45 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:25:46.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:45 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:25:46.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:45 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:25:46.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:45 smithi145 ceph-mon[95801]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-12T15:25:46.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/391343887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:25:46.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[94241]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-12T15:25:46.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[94241]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-12T15:25:46.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[94241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a5e5d11-ad42-4663-8c6b-d04e9b968c07"}]': finished 2023-07-12T15:25:46.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[94241]: osdmap e22: 4 total, 3 up, 4 in 2023-07-12T15:25:46.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:25:46.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[94241]: mgrmap e15: y(active, since 2m), standbys: x 2023-07-12T15:25:46.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:46.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[94241]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-12T15:25:46.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:25:46.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:25:46.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:25:46.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[94241]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-12T15:25:46.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/391343887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:25:46.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[102777]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-12T15:25:46.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[102777]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-12T15:25:46.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[102777]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a5e5d11-ad42-4663-8c6b-d04e9b968c07"}]': finished 2023-07-12T15:25:46.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[102777]: osdmap e22: 4 total, 3 up, 4 in 2023-07-12T15:25:46.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:25:46.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[102777]: mgrmap e15: y(active, since 2m), standbys: x 2023-07-12T15:25:46.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:46.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[102777]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-12T15:25:46.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:25:46.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:25:46.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:25:46.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[102777]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-12T15:25:46.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/391343887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:25:47.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:46 smithi145 ceph-mon[95801]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:47.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:46 smithi069 ceph-mon[94241]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:47.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:46 smithi069 ceph-mon[102777]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:49.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:48 smithi145 ceph-mon[95801]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:49.118 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:48 smithi069 ceph-mon[102777]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:49.118 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:48 smithi069 ceph-mon[94241]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:50.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:50 smithi069 ceph-mon[94241]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:50.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:50 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-12T15:25:50.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:50 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:50.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:50 smithi069 ceph-mon[102777]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:50.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:50 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-12T15:25:50.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:50 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:51.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:50 smithi145 ceph-mon[95801]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:25:51.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:50 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-12T15:25:51.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:50 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:25:51.856 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:51 smithi069 ceph-mon[94241]: Deploying daemon osd.3 on smithi069 2023-07-12T15:25:52.110 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:51 smithi069 ceph-mon[102777]: Deploying daemon osd.3 on smithi069 2023-07-12T15:25:52.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:51 smithi145 ceph-mon[95801]: Deploying daemon osd.3 on smithi069 2023-07-12T15:25:53.121 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:52 smithi069 ceph-mon[102777]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-07-12T15:25:53.121 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:52 smithi069 ceph-mon[94241]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-07-12T15:25:53.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:52 smithi145 ceph-mon[95801]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-07-12T15:25:54.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:54 smithi069 ceph-mon[102777]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-07-12T15:25:54.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:54 smithi069 ceph-mon[94241]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-07-12T15:25:55.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:54 smithi145 ceph-mon[95801]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-07-12T15:25:55.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:55 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:55.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:55 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:55.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:55 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:55.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:55 smithi069 ceph-mon[94241]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2023-07-12T15:25:55.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:55 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:55.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:55 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:55.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:55 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:55.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:55 smithi069 ceph-mon[102777]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2023-07-12T15:25:56.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:55 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:55 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:55 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:25:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:55 smithi145 ceph-mon[95801]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2023-07-12T15:25:57.655 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:57.653+0000 7ff644ff9700 1 -- 172.21.15.69:0/651812656 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff648060f40 con 0x7ff63006c690 2023-07-12T15:25:57.656 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 3 on host 'smithi069' 2023-07-12T15:25:57.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:57.654+0000 7ff64dad3700 1 -- 172.21.15.69:0/651812656 >> v2:172.21.15.69:6800/284433772 conn(0x7ff63006c690 msgr2=0x7ff63006eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:25:57.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:57.654+0000 7ff64dad3700 1 --2- 172.21.15.69:0/651812656 >> v2:172.21.15.69:6800/284433772 conn(0x7ff63006c690 0x7ff63006eb40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff63800b2e0 tx=0x7ff638005960 comp rx=0 tx=0).stop 2023-07-12T15:25:57.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:57.654+0000 7ff64dad3700 1 -- 172.21.15.69:0/651812656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6481001b0 msgr2=0x7ff648195500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:25:57.657 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:57.654+0000 7ff64dad3700 1 --2- 172.21.15.69:0/651812656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6481001b0 0x7ff648195500 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00c9c0 tx=0x7ff63c00ccd0 comp rx=0 tx=0).stop 2023-07-12T15:25:57.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:57.655+0000 7ff64dad3700 1 -- 172.21.15.69:0/651812656 shutdown_connections 2023-07-12T15:25:57.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:57.655+0000 7ff64dad3700 1 --2- 172.21.15.69:0/651812656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff648101d40 0x7ff64819cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:57.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:57.655+0000 7ff64dad3700 1 --2- 172.21.15.69:0/651812656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6481001b0 0x7ff648195500 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:57.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:57.655+0000 7ff64dad3700 1 --2- 172.21.15.69:0/651812656 >> v2:172.21.15.69:6800/284433772 conn(0x7ff63006c690 0x7ff63006eb40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:57.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:57.655+0000 7ff64dad3700 1 --2- 172.21.15.69:0/651812656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6481013b0 0x7ff648195a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:25:57.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:57.655+0000 7ff64dad3700 1 -- 172.21.15.69:0/651812656 >> 172.21.15.69:0/651812656 conn(0x7ff6480fb820 msgr2=0x7ff6481048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:25:57.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:57.655+0000 7ff64dad3700 1 -- 172.21.15.69:0/651812656 shutdown_connections 2023-07-12T15:25:57.658 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:25:57.655+0000 7ff64dad3700 1 -- 172.21.15.69:0/651812656 wait complete. 2023-07-12T15:25:58.064 DEBUG:teuthology.orchestra.run.smithi069:osd.3> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.3.service 2023-07-12T15:25:58.066 INFO:tasks.cephadm:Deploying osd.4 on smithi145 with /dev/vg_nvme/lv_4... 2023-07-12T15:25:58.067 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-12T15:25:58.095 INFO:journalctl@ceph.osd.3.smithi069.stdout:-- Logs begin at Wed 2023-07-12 15:05:12 UTC. -- 2023-07-12T15:25:58.485 INFO:journalctl@ceph.osd.3.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-3[120895]: 2023-07-12T15:25:58.128+0000 7f9a32976640 -1 osd.3 0 log_to_monitors true 2023-07-12T15:25:58.631 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:25:58.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:58 smithi145 ceph-mon[95801]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-07-12T15:25:58.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:58.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:58.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:58.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:58.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:58 smithi145 ceph-mon[95801]: from='osd.3 v2:172.21.15.69:6813/2610582937' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-12T15:25:58.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:58 smithi145 ceph-mon[95801]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-12T15:25:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[102777]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-07-12T15:25:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:58.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[102777]: from='osd.3 v2:172.21.15.69:6813/2610582937' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-12T15:25:58.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[102777]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-12T15:25:58.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[94241]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-07-12T15:25:58.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:58.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:58.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:58.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:25:58.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[94241]: from='osd.3 v2:172.21.15.69:6813/2610582937' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-12T15:25:58.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:58 smithi069 ceph-mon[94241]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-12T15:26:00.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:59 smithi145 ceph-mon[95801]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-12T15:26:00.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:59 smithi145 ceph-mon[95801]: osdmap e23: 4 total, 3 up, 4 in 2023-07-12T15:26:00.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:59 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:26:00.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:59 smithi145 ceph-mon[95801]: from='osd.3 v2:172.21.15.69:6813/2610582937' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:26:00.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:25:59 smithi145 ceph-mon[95801]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:26:00.203 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-12T15:26:00.203 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-12T15:26:00.203 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2023-07-12T15:26:00.203 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2023-07-12T15:26:00.204 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011015 s, 952 MB/s 2023-07-12T15:26:00.204 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2023-07-12T15:26:00.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:59 smithi069 ceph-mon[94241]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-12T15:26:00.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:59 smithi069 ceph-mon[94241]: osdmap e23: 4 total, 3 up, 4 in 2023-07-12T15:26:00.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:59 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:26:00.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:59 smithi069 ceph-mon[94241]: from='osd.3 v2:172.21.15.69:6813/2610582937' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:26:00.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:25:59 smithi069 ceph-mon[94241]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:26:00.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:59 smithi069 ceph-mon[102777]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-12T15:26:00.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:59 smithi069 ceph-mon[102777]: osdmap e23: 4 total, 3 up, 4 in 2023-07-12T15:26:00.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:59 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:26:00.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:59 smithi069 ceph-mon[102777]: from='osd.3 v2:172.21.15.69:6813/2610582937' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:26:00.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:25:59 smithi069 ceph-mon[102777]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2023-07-12T15:26:00.238 INFO:journalctl@ceph.osd.3.smithi069.stdout:Jul 12 15:25:59 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-3[120895]: 2023-07-12T15:25:59.844+0000 7f9a27fee700 -1 osd.3 0 waiting for initial osdmap 2023-07-12T15:26:00.238 INFO:journalctl@ceph.osd.3.smithi069.stdout:Jul 12 15:25:59 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-3[120895]: 2023-07-12T15:25:59.848+0000 7f9a205dc700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-12T15:26:00.579 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi145:vg_nvme/lv_4 2023-07-12T15:26:01.131 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:26:01.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:00 smithi069 ceph-mon[94241]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-07-12T15:26:01.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:00 smithi069 ceph-mon[94241]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2023-07-12T15:26:01.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:00 smithi069 ceph-mon[94241]: osdmap e24: 4 total, 3 up, 4 in 2023-07-12T15:26:01.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:00 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:26:01.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:00 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:26:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:00 smithi069 ceph-mon[102777]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-07-12T15:26:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:00 smithi069 ceph-mon[102777]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2023-07-12T15:26:01.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:00 smithi069 ceph-mon[102777]: osdmap e24: 4 total, 3 up, 4 in 2023-07-12T15:26:01.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:00 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:26:01.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:00 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:26:01.343 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:00 smithi145 ceph-mon[95801]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-07-12T15:26:01.344 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:00 smithi145 ceph-mon[95801]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2023-07-12T15:26:01.344 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:00 smithi145 ceph-mon[95801]: osdmap e24: 4 total, 3 up, 4 in 2023-07-12T15:26:01.344 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:00 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:26:01.344 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:00 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:26:01.954 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: purged_snaps scrub starts 2023-07-12T15:26:01.954 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: purged_snaps scrub ok 2023-07-12T15:26:01.955 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: osd.3 v2:172.21.15.69:6813/2610582937 boot 2023-07-12T15:26:01.955 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: osdmap e25: 4 total, 4 up, 4 in 2023-07-12T15:26:01.955 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:26:01.955 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: Detected new or changed devices on smithi069 2023-07-12T15:26:01.955 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:01.955 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:01.955 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:01.956 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:01.956 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:01.956 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:01.956 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: Adjusting osd_memory_target on smithi069 to 4036M 2023-07-12T15:26:01.956 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:01.956 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:01.956 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:01.956 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:01.957 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:01.957 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:01.957 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:01.957 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:01 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:02.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: purged_snaps scrub starts 2023-07-12T15:26:02.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: purged_snaps scrub ok 2023-07-12T15:26:02.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: osd.3 v2:172.21.15.69:6813/2610582937 boot 2023-07-12T15:26:02.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: osdmap e25: 4 total, 4 up, 4 in 2023-07-12T15:26:02.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:26:02.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: Detected new or changed devices on smithi069 2023-07-12T15:26:02.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:02.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:02.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:02.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:02.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:02.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:02.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: Adjusting osd_memory_target on smithi069 to 4036M 2023-07-12T15:26:02.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:02.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:02.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:02.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:02.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:02.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:02.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:02.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:02.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: purged_snaps scrub starts 2023-07-12T15:26:02.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: purged_snaps scrub ok 2023-07-12T15:26:02.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: osd.3 v2:172.21.15.69:6813/2610582937 boot 2023-07-12T15:26:02.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: osdmap e25: 4 total, 4 up, 4 in 2023-07-12T15:26:02.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:26:02.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: Detected new or changed devices on smithi069 2023-07-12T15:26:02.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:02.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:02.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:02.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:02.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:02.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:02.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: Adjusting osd_memory_target on smithi069 to 4036M 2023-07-12T15:26:02.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:02.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:02.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:02.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:02.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:02.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:02.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:02.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:01 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:02.462 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.459+0000 7f4ece453700 1 -- 172.21.15.145:0/945031716 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ec8108070 msgr2=0x7f4ec810a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:26:02.462 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.459+0000 7f4ece453700 1 --2- 172.21.15.145:0/945031716 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ec8108070 0x7f4ec810a4f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000b0a0 tx=0x7f4ec000b3b0 comp rx=0 tx=0).stop 2023-07-12T15:26:02.462 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.459+0000 7f4ece453700 1 -- 172.21.15.145:0/945031716 shutdown_connections 2023-07-12T15:26:02.462 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.459+0000 7f4ece453700 1 --2- 172.21.15.145:0/945031716 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ec8108070 0x7f4ec810a4f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:02.462 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.459+0000 7f4ece453700 1 --2- 172.21.15.145:0/945031716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec8074ab0 0x7f4ec8072f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:02.463 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.459+0000 7f4ece453700 1 --2- 172.21.15.145:0/945031716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec80734e0 0x7f4ec8073950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:02.463 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.459+0000 7f4ece453700 1 -- 172.21.15.145:0/945031716 >> 172.21.15.145:0/945031716 conn(0x7f4ec80fb820 msgr2=0x7f4ec80fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:26:02.463 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.459+0000 7f4ece453700 1 -- 172.21.15.145:0/945031716 shutdown_connections 2023-07-12T15:26:02.463 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.459+0000 7f4ece453700 1 -- 172.21.15.145:0/945031716 wait complete. 2023-07-12T15:26:02.463 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.460+0000 7f4ece453700 1 Processor -- start 2023-07-12T15:26:02.463 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.460+0000 7f4ece453700 1 -- start start 2023-07-12T15:26:02.463 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.460+0000 7f4ece453700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ec80734e0 0x7f4ec8197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:26:02.464 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.460+0000 7f4ece453700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec8074ab0 0x7f4ec8197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:26:02.464 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.460+0000 7f4ece453700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec8108070 0x7f4ec819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:26:02.464 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.460+0000 7f4ece453700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec819f5c0 con 0x7f4ec8108070 2023-07-12T15:26:02.464 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.460+0000 7f4ece453700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec819f700 con 0x7f4ec80734e0 2023-07-12T15:26:02.464 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.460+0000 7f4ece453700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec819f890 con 0x7f4ec8074ab0 2023-07-12T15:26:02.464 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.460+0000 7f4ec7fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ec80734e0 0x7f4ec8197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:26:02.464 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.460+0000 7f4ec7fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ec80734e0 0x7f4ec8197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:43818/0 (socket says 172.21.15.145:43818) 2023-07-12T15:26:02.465 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.460+0000 7f4ec7fff700 1 -- 172.21.15.145:0/4120216470 learned_addr learned my addr 172.21.15.145:0/4120216470 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:26:02.465 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.460+0000 7f4ecc9f0700 1 --2- 172.21.15.145:0/4120216470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec8108070 0x7f4ec819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:26:02.465 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.460+0000 7f4ec77fe700 1 --2- 172.21.15.145:0/4120216470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec8074ab0 0x7f4ec8197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:26:02.466 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.461+0000 7f4ec7fff700 1 -- 172.21.15.145:0/4120216470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec8074ab0 msgr2=0x7f4ec8197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:26:02.466 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.461+0000 7f4ec7fff700 1 --2- 172.21.15.145:0/4120216470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec8074ab0 0x7f4ec8197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:02.467 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.461+0000 7f4ec7fff700 1 -- 172.21.15.145:0/4120216470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec8108070 msgr2=0x7f4ec819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:26:02.467 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.461+0000 7f4ec7fff700 1 --2- 172.21.15.145:0/4120216470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec8108070 0x7f4ec819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:02.467 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.461+0000 7f4ec7fff700 1 -- 172.21.15.145:0/4120216470 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec0009d00 con 0x7f4ec80734e0 2023-07-12T15:26:02.467 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.461+0000 7f4ec77fe700 1 --2- 172.21.15.145:0/4120216470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec8074ab0 0x7f4ec8197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:26:02.467 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.461+0000 7f4ec7fff700 1 --2- 172.21.15.145:0/4120216470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ec80734e0 0x7f4ec8197a30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800b530 tx=0x7f4eb800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:26:02.467 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.461+0000 7f4ec57fa700 1 -- 172.21.15.145:0/4120216470 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8011830 con 0x7f4ec80734e0 2023-07-12T15:26:02.467 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.461+0000 7f4ec57fa700 1 -- 172.21.15.145:0/4120216470 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f4eb8011e20 con 0x7f4ec80734e0 2023-07-12T15:26:02.468 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.461+0000 7f4ece453700 1 -- 172.21.15.145:0/4120216470 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec819fb70 con 0x7f4ec80734e0 2023-07-12T15:26:02.468 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.461+0000 7f4ec57fa700 1 -- 172.21.15.145:0/4120216470 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb800f2e0 con 0x7f4ec80734e0 2023-07-12T15:26:02.468 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.461+0000 7f4ece453700 1 -- 172.21.15.145:0/4120216470 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec81a00c0 con 0x7f4ec80734e0 2023-07-12T15:26:02.468 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.462+0000 7f4ece453700 1 -- 172.21.15.145:0/4120216470 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec81067b0 con 0x7f4ec80734e0 2023-07-12T15:26:02.468 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.463+0000 7f4ec57fa700 1 -- 172.21.15.145:0/4120216470 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 15) v1 ==== 89784+0+0 (secure 0 0 0) 0x7f4eb8011990 con 0x7f4ec80734e0 2023-07-12T15:26:02.468 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.463+0000 7f4ec57fa700 1 --2- 172.21.15.145:0/4120216470 >> v2:172.21.15.69:6800/284433772 conn(0x7f4eb006c720 0x7f4eb006ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:26:02.468 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.463+0000 7f4ec57fa700 1 -- 172.21.15.145:0/4120216470 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 2803+0+0 (secure 0 0 0) 0x7f4eb808a790 con 0x7f4ec80734e0 2023-07-12T15:26:02.469 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.463+0000 7f4ec77fe700 1 --2- 172.21.15.145:0/4120216470 >> v2:172.21.15.69:6800/284433772 conn(0x7f4eb006c720 0x7f4eb006ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:26:02.469 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.464+0000 7f4ec77fe700 1 --2- 172.21.15.145:0/4120216470 >> v2:172.21.15.69:6800/284433772 conn(0x7f4eb006c720 0x7f4eb006ebd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0099d0 tx=0x7f4ebc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:26:02.469 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.468+0000 7f4ec57fa700 1 -- 172.21.15.145:0/4120216470 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4eb8055950 con 0x7f4ec80734e0 2023-07-12T15:26:02.661 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:02.660+0000 7f4ece453700 1 -- 172.21.15.145:0/4120216470 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ec8060f40 con 0x7f4eb006c720 2023-07-12T15:26:03.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:02 smithi069 ceph-mon[94241]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:26:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:02 smithi069 ceph-mon[94241]: osdmap e26: 4 total, 4 up, 4 in 2023-07-12T15:26:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:26:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:26:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:03.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:02 smithi069 ceph-mon[102777]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:26:03.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:02 smithi069 ceph-mon[102777]: osdmap e26: 4 total, 4 up, 4 in 2023-07-12T15:26:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:02 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:26:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:02 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:26:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:02 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:03.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:02 smithi145 ceph-mon[95801]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-12T15:26:03.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:02 smithi145 ceph-mon[95801]: osdmap e26: 4 total, 4 up, 4 in 2023-07-12T15:26:03.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:02 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:26:03.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:02 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:26:03.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:02 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:04.012 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:03 smithi145 ceph-mon[95801]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:26:04.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:03 smithi069 ceph-mon[94241]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:26:04.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:03 smithi069 ceph-mon[102777]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:26:05.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:04 smithi069 ceph-mon[102777]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:05.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:04 smithi069 ceph-mon[94241]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:05.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:04 smithi145 ceph-mon[95801]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:06.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/3515300611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "672b53e3-659e-408b-98e5-a3cbaf549b70"}]: dispatch 2023-07-12T15:26:06.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[94241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "672b53e3-659e-408b-98e5-a3cbaf549b70"}]: dispatch 2023-07-12T15:26:06.297 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[94241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "672b53e3-659e-408b-98e5-a3cbaf549b70"}]': finished 2023-07-12T15:26:06.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[94241]: osdmap e27: 5 total, 4 up, 5 in 2023-07-12T15:26:06.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:06.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[94241]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:06.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/3937307069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:26:06.299 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/3515300611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "672b53e3-659e-408b-98e5-a3cbaf549b70"}]: dispatch 2023-07-12T15:26:06.299 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[102777]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "672b53e3-659e-408b-98e5-a3cbaf549b70"}]: dispatch 2023-07-12T15:26:06.299 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[102777]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "672b53e3-659e-408b-98e5-a3cbaf549b70"}]': finished 2023-07-12T15:26:06.299 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[102777]: osdmap e27: 5 total, 4 up, 5 in 2023-07-12T15:26:06.300 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:06.300 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[102777]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:06.300 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/3937307069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:26:06.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/3515300611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "672b53e3-659e-408b-98e5-a3cbaf549b70"}]: dispatch 2023-07-12T15:26:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:05 smithi145 ceph-mon[95801]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "672b53e3-659e-408b-98e5-a3cbaf549b70"}]: dispatch 2023-07-12T15:26:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:05 smithi145 ceph-mon[95801]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "672b53e3-659e-408b-98e5-a3cbaf549b70"}]': finished 2023-07-12T15:26:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:05 smithi145 ceph-mon[95801]: osdmap e27: 5 total, 4 up, 5 in 2023-07-12T15:26:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:05 smithi145 ceph-mon[95801]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/3937307069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:26:08.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:08 smithi145 ceph-mon[95801]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:08.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:08 smithi069 ceph-mon[94241]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:08.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:08 smithi069 ceph-mon[102777]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:10.810 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:10 smithi145 ceph-mon[95801]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:10.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:10 smithi069 ceph-mon[94241]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:10.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:10 smithi069 ceph-mon[102777]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:11.802 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:11 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-12T15:26:11.802 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:11 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:11.803 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:11 smithi145 ceph-mon[95801]: Deploying daemon osd.4 on smithi145 2023-07-12T15:26:11.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:11 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-12T15:26:11.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:11 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:11.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:11 smithi069 ceph-mon[94241]: Deploying daemon osd.4 on smithi145 2023-07-12T15:26:11.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:11 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-12T15:26:11.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:11 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:11.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:11 smithi069 ceph-mon[102777]: Deploying daemon osd.4 on smithi145 2023-07-12T15:26:12.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:12 smithi145 ceph-mon[95801]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:12.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:12 smithi069 ceph-mon[102777]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:12.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:12 smithi069 ceph-mon[94241]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:14.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:14 smithi145 ceph-mon[95801]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:14.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:14 smithi069 ceph-mon[102777]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:14.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:14 smithi069 ceph-mon[94241]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:16.327 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:16 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:16.327 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:16 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:16.327 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:16 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:16.327 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:16 smithi145 ceph-mon[95801]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:16.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:16 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:16.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:16 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:16.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:16 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:16.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:16 smithi069 ceph-mon[94241]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:16.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:16 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:16.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:16 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:16.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:16 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:16.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:16 smithi069 ceph-mon[102777]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:18.221 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 4 on host 'smithi145' 2023-07-12T15:26:18.221 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:18.219+0000 7f4ec57fa700 1 -- 172.21.15.145:0/4120216470 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4ec8060f40 con 0x7f4eb006c720 2023-07-12T15:26:18.223 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:18.221+0000 7f4ece453700 1 -- 172.21.15.145:0/4120216470 >> v2:172.21.15.69:6800/284433772 conn(0x7f4eb006c720 msgr2=0x7f4eb006ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:26:18.223 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:18.221+0000 7f4ece453700 1 --2- 172.21.15.145:0/4120216470 >> v2:172.21.15.69:6800/284433772 conn(0x7f4eb006c720 0x7f4eb006ebd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0099d0 tx=0x7f4ebc008040 comp rx=0 tx=0).stop 2023-07-12T15:26:18.223 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:18.221+0000 7f4ece453700 1 -- 172.21.15.145:0/4120216470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ec80734e0 msgr2=0x7f4ec8197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:26:18.223 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:18.221+0000 7f4ece453700 1 --2- 172.21.15.145:0/4120216470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ec80734e0 0x7f4ec8197a30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800b530 tx=0x7f4eb800b840 comp rx=0 tx=0).stop 2023-07-12T15:26:18.223 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:18.221+0000 7f4ece453700 1 -- 172.21.15.145:0/4120216470 shutdown_connections 2023-07-12T15:26:18.223 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:18.221+0000 7f4ece453700 1 --2- 172.21.15.145:0/4120216470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec8108070 0x7f4ec819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:18.224 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:18.221+0000 7f4ece453700 1 --2- 172.21.15.145:0/4120216470 >> v2:172.21.15.69:6800/284433772 conn(0x7f4eb006c720 0x7f4eb006ebd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:18.224 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:18.221+0000 7f4ece453700 1 --2- 172.21.15.145:0/4120216470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ec80734e0 0x7f4ec8197a30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:18.224 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:18.221+0000 7f4ece453700 1 --2- 172.21.15.145:0/4120216470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ec8074ab0 0x7f4ec8197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:18.224 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:18.221+0000 7f4ece453700 1 -- 172.21.15.145:0/4120216470 >> 172.21.15.145:0/4120216470 conn(0x7f4ec80fb820 msgr2=0x7f4ec80fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:26:18.224 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:18.221+0000 7f4ece453700 1 -- 172.21.15.145:0/4120216470 shutdown_connections 2023-07-12T15:26:18.224 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:18.221+0000 7f4ece453700 1 -- 172.21.15.145:0/4120216470 wait complete. 2023-07-12T15:26:18.609 DEBUG:teuthology.orchestra.run.smithi145:osd.4> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.4.service 2023-07-12T15:26:18.612 INFO:tasks.cephadm:Deploying osd.5 on smithi145 with /dev/vg_nvme/lv_3... 2023-07-12T15:26:18.615 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-12T15:26:18.643 INFO:journalctl@ceph.osd.4.smithi145.stdout:-- Logs begin at Wed 2023-07-12 15:04:57 UTC. -- 2023-07-12T15:26:18.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:18 smithi145 ceph-mon[95801]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:18 smithi145 ceph-mon[95801]: from='osd.4 v2:172.21.15.145:6800/184109944' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-12T15:26:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:18 smithi145 ceph-mon[95801]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-12T15:26:18.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:18 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:18.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[94241]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:18.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:18.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:18.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[94241]: from='osd.4 v2:172.21.15.145:6800/184109944' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-12T15:26:18.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[94241]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-12T15:26:18.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:18.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[102777]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:18.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:18.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:18.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[102777]: from='osd.4 v2:172.21.15.145:6800/184109944' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-12T15:26:18.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[102777]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-12T15:26:18.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:18.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:18 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:19.328 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:26:20.351 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:26:20 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:26:20.037+0000 7f55d50af700 -1 osd.4 0 waiting for initial osdmap 2023-07-12T15:26:20.351 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:26:20 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:26:20.043+0000 7f55d0ea4700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-12T15:26:20.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:20 smithi145 ceph-mon[95801]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-12T15:26:20.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:20 smithi145 ceph-mon[95801]: osdmap e28: 5 total, 4 up, 5 in 2023-07-12T15:26:20.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:20 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:20.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:20 smithi145 ceph-mon[95801]: from='osd.4 v2:172.21.15.145:6800/184109944' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:26:20.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:20 smithi145 ceph-mon[95801]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:26:20.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:20 smithi145 ceph-mon[95801]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:20.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:20 smithi069 ceph-mon[94241]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-12T15:26:20.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:20 smithi069 ceph-mon[94241]: osdmap e28: 5 total, 4 up, 5 in 2023-07-12T15:26:20.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:20 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:20.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:20 smithi069 ceph-mon[94241]: from='osd.4 v2:172.21.15.145:6800/184109944' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:26:20.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:20 smithi069 ceph-mon[94241]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:26:20.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:20 smithi069 ceph-mon[94241]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:20.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:20 smithi069 ceph-mon[102777]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-12T15:26:20.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:20 smithi069 ceph-mon[102777]: osdmap e28: 5 total, 4 up, 5 in 2023-07-12T15:26:20.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:20 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:20.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:20 smithi069 ceph-mon[102777]: from='osd.4 v2:172.21.15.145:6800/184109944' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:26:20.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:20 smithi069 ceph-mon[102777]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:26:20.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:20 smithi069 ceph-mon[102777]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-12T15:26:21.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:21 smithi145 ceph-mon[95801]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2023-07-12T15:26:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:21 smithi145 ceph-mon[95801]: osdmap e29: 5 total, 4 up, 5 in 2023-07-12T15:26:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:21 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:21 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:21.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:21 smithi069 ceph-mon[94241]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2023-07-12T15:26:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:21 smithi069 ceph-mon[94241]: osdmap e29: 5 total, 4 up, 5 in 2023-07-12T15:26:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:21 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:21 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:21.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:21 smithi069 ceph-mon[102777]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2023-07-12T15:26:21.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:21 smithi069 ceph-mon[102777]: osdmap e29: 5 total, 4 up, 5 in 2023-07-12T15:26:21.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:21.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:21.570 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-12T15:26:21.612 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-12T15:26:21.612 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2023-07-12T15:26:21.612 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2023-07-12T15:26:21.612 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130164 s, 806 MB/s 2023-07-12T15:26:21.613 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2023-07-12T15:26:22.030 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi145:vg_nvme/lv_3 2023-07-12T15:26:22.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: purged_snaps scrub starts 2023-07-12T15:26:22.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: purged_snaps scrub ok 2023-07-12T15:26:22.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: osd.4 v2:172.21.15.145:6800/184109944 boot 2023-07-12T15:26:22.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: osdmap e30: 5 total, 5 up, 5 in 2023-07-12T15:26:22.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:22.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: Detected new or changed devices on smithi145 2023-07-12T15:26:22.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:22.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:22.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:22.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: Adjusting osd_memory_target on smithi145 to 17168M 2023-07-12T15:26:22.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:22.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:22.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:22.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:22.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:22.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:22 smithi145 ceph-mon[95801]: osdmap e31: 5 total, 5 up, 5 in 2023-07-12T15:26:22.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: purged_snaps scrub starts 2023-07-12T15:26:22.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: purged_snaps scrub ok 2023-07-12T15:26:22.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: osd.4 v2:172.21.15.145:6800/184109944 boot 2023-07-12T15:26:22.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: osdmap e30: 5 total, 5 up, 5 in 2023-07-12T15:26:22.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:22.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: Detected new or changed devices on smithi145 2023-07-12T15:26:22.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:22.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:22.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:22.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: Adjusting osd_memory_target on smithi145 to 17168M 2023-07-12T15:26:22.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:22.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[94241]: osdmap e31: 5 total, 5 up, 5 in 2023-07-12T15:26:22.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: purged_snaps scrub starts 2023-07-12T15:26:22.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: purged_snaps scrub ok 2023-07-12T15:26:22.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: osd.4 v2:172.21.15.145:6800/184109944 boot 2023-07-12T15:26:22.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: osdmap e30: 5 total, 5 up, 5 in 2023-07-12T15:26:22.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:26:22.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: Detected new or changed devices on smithi145 2023-07-12T15:26:22.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:22.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:22.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:22.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: Adjusting osd_memory_target on smithi145 to 17168M 2023-07-12T15:26:22.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:22.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:22.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:22.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:22.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:22.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:22 smithi069 ceph-mon[102777]: osdmap e31: 5 total, 5 up, 5 in 2023-07-12T15:26:22.582 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:26:23.949 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.946+0000 7f9e14bf4700 1 -- 172.21.15.145:0/914519475 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e10101430 msgr2=0x7f9e101018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:26:23.950 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.946+0000 7f9e14bf4700 1 --2- 172.21.15.145:0/914519475 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e10101430 0x7f9e101018a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009990 tx=0x7f9e00009ca0 comp rx=0 tx=0).stop 2023-07-12T15:26:23.950 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.947+0000 7f9e14bf4700 1 -- 172.21.15.145:0/914519475 shutdown_connections 2023-07-12T15:26:23.950 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.947+0000 7f9e14bf4700 1 --2- 172.21.15.145:0/914519475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e10101de0 0x7f9e10108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:23.950 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.947+0000 7f9e14bf4700 1 --2- 172.21.15.145:0/914519475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e10100190 0x7f9e101005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:23.950 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.947+0000 7f9e14bf4700 1 --2- 172.21.15.145:0/914519475 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e10101430 0x7f9e101018a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:23.950 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.947+0000 7f9e14bf4700 1 -- 172.21.15.145:0/914519475 >> 172.21.15.145:0/914519475 conn(0x7f9e100fb7e0 msgr2=0x7f9e100fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:26:23.950 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.947+0000 7f9e14bf4700 1 -- 172.21.15.145:0/914519475 shutdown_connections 2023-07-12T15:26:23.951 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.947+0000 7f9e14bf4700 1 -- 172.21.15.145:0/914519475 wait complete. 2023-07-12T15:26:23.952 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.947+0000 7f9e14bf4700 1 Processor -- start 2023-07-12T15:26:23.952 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.947+0000 7f9e14bf4700 1 -- start start 2023-07-12T15:26:23.952 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.947+0000 7f9e14bf4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e10100190 0x7f9e10195700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:26:23.953 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.947+0000 7f9e14bf4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e10101430 0x7f9e10195c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:26:23.953 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e14bf4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e10101de0 0x7f9e1019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:26:23.953 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e14bf4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1019d290 con 0x7f9e10101430 2023-07-12T15:26:23.953 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e14bf4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1019d3d0 con 0x7f9e10100190 2023-07-12T15:26:23.953 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e14bf4700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1019d560 con 0x7f9e10101de0 2023-07-12T15:26:23.953 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e0e59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e10100190 0x7f9e10195700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:26:23.953 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e0e59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e10100190 0x7f9e10195700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:43902/0 (socket says 172.21.15.145:43902) 2023-07-12T15:26:23.954 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e0e59c700 1 -- 172.21.15.145:0/106938232 learned_addr learned my addr 172.21.15.145:0/106938232 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:26:23.954 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e0dd9b700 1 --2- 172.21.15.145:0/106938232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e10101430 0x7f9e10195c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:26:23.954 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e0ed9d700 1 --2- 172.21.15.145:0/106938232 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e10101de0 0x7f9e1019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:26:23.954 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e0e59c700 1 -- 172.21.15.145:0/106938232 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e10101de0 msgr2=0x7f9e1019ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:26:23.954 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e0e59c700 1 --2- 172.21.15.145:0/106938232 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e10101de0 0x7f9e1019ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:23.954 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e0e59c700 1 -- 172.21.15.145:0/106938232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e10101430 msgr2=0x7f9e10195c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:26:23.954 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e0e59c700 1 --2- 172.21.15.145:0/106938232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e10101430 0x7f9e10195c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:23.955 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e0e59c700 1 -- 172.21.15.145:0/106938232 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e000095f0 con 0x7f9e10100190 2023-07-12T15:26:23.955 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e0e59c700 1 --2- 172.21.15.145:0/106938232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e10100190 0x7f9e10195700 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400b530 tx=0x7f9e0400b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:26:23.955 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9dff7fe700 1 -- 172.21.15.145:0/106938232 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04011830 con 0x7f9e10100190 2023-07-12T15:26:23.955 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e14bf4700 1 -- 172.21.15.145:0/106938232 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1019d840 con 0x7f9e10100190 2023-07-12T15:26:23.955 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9dff7fe700 1 -- 172.21.15.145:0/106938232 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f9e04011e20 con 0x7f9e10100190 2023-07-12T15:26:23.955 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9dff7fe700 1 -- 172.21.15.145:0/106938232 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04010dd0 con 0x7f9e10100190 2023-07-12T15:26:23.955 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.948+0000 7f9e14bf4700 1 -- 172.21.15.145:0/106938232 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1019ddc0 con 0x7f9e10100190 2023-07-12T15:26:23.955 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.949+0000 7f9e14bf4700 1 -- 172.21.15.145:0/106938232 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1004e800 con 0x7f9e10100190 2023-07-12T15:26:23.956 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.950+0000 7f9dff7fe700 1 -- 172.21.15.145:0/106938232 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 15) v1 ==== 89784+0+0 (secure 0 0 0) 0x7f9e04011990 con 0x7f9e10100190 2023-07-12T15:26:23.959 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.950+0000 7f9dff7fe700 1 --2- 172.21.15.145:0/106938232 >> v2:172.21.15.69:6800/284433772 conn(0x7f9df806c720 0x7f9df806ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:26:23.959 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.950+0000 7f9dff7fe700 1 -- 172.21.15.145:0/106938232 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 3234+0+0 (secure 0 0 0) 0x7f9e0408a970 con 0x7f9e10100190 2023-07-12T15:26:23.959 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.950+0000 7f9e0dd9b700 1 --2- 172.21.15.145:0/106938232 >> v2:172.21.15.69:6800/284433772 conn(0x7f9df806c720 0x7f9df806ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:26:23.959 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.951+0000 7f9e0dd9b700 1 --2- 172.21.15.145:0/106938232 >> v2:172.21.15.69:6800/284433772 conn(0x7f9df806c720 0x7f9df806ebd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9e000095c0 tx=0x7f9e00009530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:26:23.959 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:23.955+0000 7f9dff7fe700 1 -- 172.21.15.145:0/106938232 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9e04055900 con 0x7f9e10100190 2023-07-12T15:26:24.153 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:24.150+0000 7f9e14bf4700 1 -- 172.21.15.145:0/106938232 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e1019dfe0 con 0x7f9df806c720 2023-07-12T15:26:24.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:24 smithi145 ceph-mon[95801]: osdmap e32: 5 total, 5 up, 5 in 2023-07-12T15:26:24.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:24 smithi145 ceph-mon[95801]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:24.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:24 smithi069 ceph-mon[94241]: osdmap e32: 5 total, 5 up, 5 in 2023-07-12T15:26:24.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:24 smithi069 ceph-mon[94241]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:24.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:24 smithi069 ceph-mon[102777]: osdmap e32: 5 total, 5 up, 5 in 2023-07-12T15:26:24.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:24 smithi069 ceph-mon[102777]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:25.291 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:25 smithi145 ceph-mon[95801]: osdmap e33: 5 total, 5 up, 5 in 2023-07-12T15:26:25.292 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:25 smithi145 ceph-mon[95801]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:26:25.292 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:25 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:26:25.292 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:25 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:26:25.292 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:25 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:25.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:25 smithi069 ceph-mon[94241]: osdmap e33: 5 total, 5 up, 5 in 2023-07-12T15:26:25.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:25 smithi069 ceph-mon[94241]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:26:25.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:25 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:26:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:25 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:26:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:25 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:25.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:25 smithi069 ceph-mon[102777]: osdmap e33: 5 total, 5 up, 5 in 2023-07-12T15:26:25.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:25 smithi069 ceph-mon[102777]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:26:25.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:25 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:26:25.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:25 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:26:25.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:25 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:26.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:26 smithi145 ceph-mon[95801]: osdmap e34: 5 total, 5 up, 5 in 2023-07-12T15:26:26.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:26 smithi145 ceph-mon[95801]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-12T15:26:26.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:26 smithi069 ceph-mon[94241]: osdmap e34: 5 total, 5 up, 5 in 2023-07-12T15:26:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:26 smithi069 ceph-mon[94241]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-12T15:26:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:26 smithi069 ceph-mon[102777]: osdmap e34: 5 total, 5 up, 5 in 2023-07-12T15:26:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:26 smithi069 ceph-mon[102777]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-12T15:26:27.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/1773572968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "975b3691-5232-4774-9b74-087ae97745a6"}]: dispatch 2023-07-12T15:26:27.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:27 smithi145 ceph-mon[95801]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "975b3691-5232-4774-9b74-087ae97745a6"}]: dispatch 2023-07-12T15:26:27.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:27 smithi145 ceph-mon[95801]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "975b3691-5232-4774-9b74-087ae97745a6"}]': finished 2023-07-12T15:26:27.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:27 smithi145 ceph-mon[95801]: osdmap e35: 6 total, 5 up, 6 in 2023-07-12T15:26:27.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:27 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:27.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/1773572968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "975b3691-5232-4774-9b74-087ae97745a6"}]: dispatch 2023-07-12T15:26:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:27 smithi069 ceph-mon[94241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "975b3691-5232-4774-9b74-087ae97745a6"}]: dispatch 2023-07-12T15:26:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:27 smithi069 ceph-mon[94241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "975b3691-5232-4774-9b74-087ae97745a6"}]': finished 2023-07-12T15:26:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:27 smithi069 ceph-mon[94241]: osdmap e35: 6 total, 5 up, 6 in 2023-07-12T15:26:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:27 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:27.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/1773572968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "975b3691-5232-4774-9b74-087ae97745a6"}]: dispatch 2023-07-12T15:26:27.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:27 smithi069 ceph-mon[102777]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "975b3691-5232-4774-9b74-087ae97745a6"}]: dispatch 2023-07-12T15:26:27.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:27 smithi069 ceph-mon[102777]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "975b3691-5232-4774-9b74-087ae97745a6"}]': finished 2023-07-12T15:26:27.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:27 smithi069 ceph-mon[102777]: osdmap e35: 6 total, 5 up, 6 in 2023-07-12T15:26:27.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:27 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:28.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/2455152018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:26:28.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:28 smithi145 ceph-mon[95801]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 130 KiB/s, 0 objects/s recovering 2023-07-12T15:26:28.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/2455152018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:26:28.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:28 smithi069 ceph-mon[94241]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 130 KiB/s, 0 objects/s recovering 2023-07-12T15:26:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/2455152018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:26:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:28 smithi069 ceph-mon[102777]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 130 KiB/s, 0 objects/s recovering 2023-07-12T15:26:30.760 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:30 smithi145 ceph-mon[95801]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-12T15:26:30.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:30 smithi069 ceph-mon[94241]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-12T15:26:30.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:30 smithi069 ceph-mon[102777]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-12T15:26:32.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:32 smithi145 ceph-mon[95801]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-07-12T15:26:32.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:32 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-12T15:26:32.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:32 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:32.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:32 smithi069 ceph-mon[94241]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-07-12T15:26:32.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:32 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-12T15:26:32.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:32 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:32.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:32 smithi069 ceph-mon[102777]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-07-12T15:26:32.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:32 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-12T15:26:32.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:32 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:33.531 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:33 smithi145 ceph-mon[95801]: Deploying daemon osd.5 on smithi145 2023-07-12T15:26:33.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:33 smithi069 ceph-mon[94241]: Deploying daemon osd.5 on smithi145 2023-07-12T15:26:33.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:33 smithi069 ceph-mon[102777]: Deploying daemon osd.5 on smithi145 2023-07-12T15:26:34.837 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:34 smithi145 ceph-mon[95801]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:34.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:34 smithi069 ceph-mon[94241]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:34.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:34 smithi069 ceph-mon[102777]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:35 smithi145 ceph-mon[95801]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:36.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:35 smithi069 ceph-mon[94241]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:36.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:35 smithi069 ceph-mon[102777]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:37.942 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:37.943 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:37.943 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:38.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:38.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:38.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:38.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:38.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:38.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:39.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:38 smithi145 ceph-mon[95801]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:39.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:38 smithi069 ceph-mon[94241]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:38 smithi069 ceph-mon[102777]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:39.811 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 5 on host 'smithi145' 2023-07-12T15:26:39.811 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:39.809+0000 7f9dff7fe700 1 -- 172.21.15.145:0/106938232 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9e1019dfe0 con 0x7f9df806c720 2023-07-12T15:26:39.813 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:39.811+0000 7f9e14bf4700 1 -- 172.21.15.145:0/106938232 >> v2:172.21.15.69:6800/284433772 conn(0x7f9df806c720 msgr2=0x7f9df806ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:26:39.813 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:39.811+0000 7f9e14bf4700 1 --2- 172.21.15.145:0/106938232 >> v2:172.21.15.69:6800/284433772 conn(0x7f9df806c720 0x7f9df806ebd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9e000095c0 tx=0x7f9e00009530 comp rx=0 tx=0).stop 2023-07-12T15:26:39.813 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:39.811+0000 7f9e14bf4700 1 -- 172.21.15.145:0/106938232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e10100190 msgr2=0x7f9e10195700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:26:39.813 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:39.811+0000 7f9e14bf4700 1 --2- 172.21.15.145:0/106938232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e10100190 0x7f9e10195700 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400b530 tx=0x7f9e0400b8f0 comp rx=0 tx=0).stop 2023-07-12T15:26:39.814 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:39.811+0000 7f9e14bf4700 1 -- 172.21.15.145:0/106938232 shutdown_connections 2023-07-12T15:26:39.814 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:39.811+0000 7f9e14bf4700 1 --2- 172.21.15.145:0/106938232 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e10101de0 0x7f9e1019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:39.814 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:39.811+0000 7f9e14bf4700 1 --2- 172.21.15.145:0/106938232 >> v2:172.21.15.69:6800/284433772 conn(0x7f9df806c720 0x7f9df806ebd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:39.814 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:39.811+0000 7f9e14bf4700 1 --2- 172.21.15.145:0/106938232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e10100190 0x7f9e10195700 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:39.814 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:39.811+0000 7f9e14bf4700 1 --2- 172.21.15.145:0/106938232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e10101430 0x7f9e10195c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:39.815 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:39.811+0000 7f9e14bf4700 1 -- 172.21.15.145:0/106938232 >> 172.21.15.145:0/106938232 conn(0x7f9e100fb7e0 msgr2=0x7f9e100fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:26:39.815 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:39.812+0000 7f9e14bf4700 1 -- 172.21.15.145:0/106938232 shutdown_connections 2023-07-12T15:26:39.815 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:39.812+0000 7f9e14bf4700 1 -- 172.21.15.145:0/106938232 wait complete. 2023-07-12T15:26:40.271 DEBUG:teuthology.orchestra.run.smithi145:osd.5> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.5.service 2023-07-12T15:26:40.274 INFO:tasks.cephadm:Deploying osd.6 on smithi145 with /dev/vg_nvme/lv_2... 2023-07-12T15:26:40.274 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-12T15:26:40.306 INFO:journalctl@ceph.osd.5.smithi145.stdout:-- Logs begin at Wed 2023-07-12 15:04:57 UTC. -- 2023-07-12T15:26:40.329 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.330 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.330 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:40.330 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:40.330 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.330 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:40 smithi145 ceph-mon[95801]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:40.331 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.331 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.331 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:40.331 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:40 smithi145 ceph-mon[95801]: from='osd.5 v2:172.21.15.145:6804/363217135' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-12T15:26:40.331 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:40 smithi145 ceph-mon[95801]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-12T15:26:40.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:40.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:40.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[94241]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[94241]: from='osd.5 v2:172.21.15.145:6804/363217135' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-12T15:26:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[94241]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-12T15:26:40.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:40.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:40.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[102777]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:40.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:40.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:26:40.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[102777]: from='osd.5 v2:172.21.15.145:6804/363217135' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-12T15:26:40.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:40 smithi069 ceph-mon[102777]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-12T15:26:41.032 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:26:42.101 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:26:41 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:26:41.943+0000 7fa25d2eb700 -1 osd.5 0 waiting for initial osdmap 2023-07-12T15:26:42.101 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:26:41 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:26:41.950+0000 7fa2570dc700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-12T15:26:42.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:41 smithi145 ceph-mon[95801]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-12T15:26:42.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:41 smithi145 ceph-mon[95801]: osdmap e36: 6 total, 5 up, 6 in 2023-07-12T15:26:42.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:41 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:42.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:41 smithi145 ceph-mon[95801]: from='osd.5 v2:172.21.15.145:6804/363217135' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:26:42.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:41 smithi145 ceph-mon[95801]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:26:42.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:41 smithi145 ceph-mon[95801]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:42.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:41 smithi069 ceph-mon[94241]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-12T15:26:42.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:41 smithi069 ceph-mon[94241]: osdmap e36: 6 total, 5 up, 6 in 2023-07-12T15:26:42.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:41 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:42.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:41 smithi069 ceph-mon[94241]: from='osd.5 v2:172.21.15.145:6804/363217135' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:26:42.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:41 smithi069 ceph-mon[94241]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:26:42.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:41 smithi069 ceph-mon[94241]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:42.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:41 smithi069 ceph-mon[102777]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-12T15:26:42.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:41 smithi069 ceph-mon[102777]: osdmap e36: 6 total, 5 up, 6 in 2023-07-12T15:26:42.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:41 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:42.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:41 smithi069 ceph-mon[102777]: from='osd.5 v2:172.21.15.145:6804/363217135' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:26:42.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:41 smithi069 ceph-mon[102777]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:26:42.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:41 smithi069 ceph-mon[102777]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-12T15:26:43.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[94241]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2023-07-12T15:26:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[94241]: osdmap e37: 6 total, 5 up, 6 in 2023-07-12T15:26:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:43.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:43.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[94241]: osd.5 v2:172.21.15.145:6804/363217135 boot 2023-07-12T15:26:43.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[94241]: osdmap e38: 6 total, 6 up, 6 in 2023-07-12T15:26:43.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:43.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[102777]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2023-07-12T15:26:43.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[102777]: osdmap e37: 6 total, 5 up, 6 in 2023-07-12T15:26:43.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:43.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:43.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[102777]: osd.5 v2:172.21.15.145:6804/363217135 boot 2023-07-12T15:26:43.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[102777]: osdmap e38: 6 total, 6 up, 6 in 2023-07-12T15:26:43.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:42 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:43.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:42 smithi145 ceph-mon[95801]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2023-07-12T15:26:43.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:42 smithi145 ceph-mon[95801]: osdmap e37: 6 total, 5 up, 6 in 2023-07-12T15:26:43.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:42 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:43.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:42 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:43.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:42 smithi145 ceph-mon[95801]: osd.5 v2:172.21.15.145:6804/363217135 boot 2023-07-12T15:26:43.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:42 smithi145 ceph-mon[95801]: osdmap e38: 6 total, 6 up, 6 in 2023-07-12T15:26:43.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:42 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:26:43.645 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-12T15:26:43.646 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-12T15:26:43.646 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2023-07-12T15:26:43.646 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2023-07-12T15:26:43.646 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011149 s, 941 MB/s 2023-07-12T15:26:43.646 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2023-07-12T15:26:44.091 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi145:vg_nvme/lv_2 2023-07-12T15:26:44.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: purged_snaps scrub starts 2023-07-12T15:26:44.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: purged_snaps scrub ok 2023-07-12T15:26:44.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: Detected new or changed devices on smithi145 2023-07-12T15:26:44.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:44.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:44.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:44.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:44.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: Adjusting osd_memory_target on smithi145 to 8584M 2023-07-12T15:26:44.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:44.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:44.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:44.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:44.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:26:44.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:44 smithi145 ceph-mon[95801]: osdmap e39: 6 total, 6 up, 6 in 2023-07-12T15:26:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: purged_snaps scrub starts 2023-07-12T15:26:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: purged_snaps scrub ok 2023-07-12T15:26:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: Detected new or changed devices on smithi145 2023-07-12T15:26:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:44.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:44.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: Adjusting osd_memory_target on smithi145 to 8584M 2023-07-12T15:26:44.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:44.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:44.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:44.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:44.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:26:44.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[94241]: osdmap e39: 6 total, 6 up, 6 in 2023-07-12T15:26:44.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: purged_snaps scrub starts 2023-07-12T15:26:44.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: purged_snaps scrub ok 2023-07-12T15:26:44.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: Detected new or changed devices on smithi145 2023-07-12T15:26:44.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:44.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:44.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:44.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:26:44.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: Adjusting osd_memory_target on smithi145 to 8584M 2023-07-12T15:26:44.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:44.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:44.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:26:44.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:26:44.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:26:44.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:44 smithi069 ceph-mon[102777]: osdmap e39: 6 total, 6 up, 6 in 2023-07-12T15:26:44.686 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:26:46.035 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.029+0000 7f0107da4700 1 -- 172.21.15.145:0/2661520077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0100105ea0 msgr2=0x7f01001082a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:26:46.035 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.029+0000 7f0107da4700 1 --2- 172.21.15.145:0/2661520077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0100105ea0 0x7f01001082a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f00f800b0a0 tx=0x7f00f800b3b0 comp rx=0 tx=0).stop 2023-07-12T15:26:46.036 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.029+0000 7f0107da4700 1 -- 172.21.15.145:0/2661520077 shutdown_connections 2023-07-12T15:26:46.036 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.029+0000 7f0107da4700 1 --2- 172.21.15.145:0/2661520077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0100105ea0 0x7f01001082a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:46.036 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.029+0000 7f0107da4700 1 --2- 172.21.15.145:0/2661520077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0100100c60 0x7f0100103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:46.036 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.029+0000 7f0107da4700 1 --2- 172.21.15.145:0/2661520077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0100103580 0x7f0100105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:46.037 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.029+0000 7f0107da4700 1 -- 172.21.15.145:0/2661520077 >> 172.21.15.145:0/2661520077 conn(0x7f01000fa6a0 msgr2=0x7f01000fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:26:46.037 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.029+0000 7f0107da4700 1 -- 172.21.15.145:0/2661520077 shutdown_connections 2023-07-12T15:26:46.038 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.029+0000 7f0107da4700 1 -- 172.21.15.145:0/2661520077 wait complete. 2023-07-12T15:26:46.038 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0107da4700 1 Processor -- start 2023-07-12T15:26:46.038 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0107da4700 1 -- start start 2023-07-12T15:26:46.039 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0107da4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0100100c60 0x7f01001956f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:26:46.039 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0107da4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0100103580 0x7f0100195c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:26:46.039 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0107da4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0100105ea0 0x7f010019ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:26:46.039 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0107da4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010019d280 con 0x7f0100103580 2023-07-12T15:26:46.039 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0107da4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010019d3c0 con 0x7f0100100c60 2023-07-12T15:26:46.039 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0107da4700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010019d550 con 0x7f0100105ea0 2023-07-12T15:26:46.040 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0105b40700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0100100c60 0x7f01001956f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:26:46.040 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0105b40700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0100100c60 0x7f01001956f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:43994/0 (socket says 172.21.15.145:43994) 2023-07-12T15:26:46.040 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0105b40700 1 -- 172.21.15.145:0/2240509801 learned_addr learned my addr 172.21.15.145:0/2240509801 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:26:46.040 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0106341700 1 --2- 172.21.15.145:0/2240509801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0100105ea0 0x7f010019ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:26:46.040 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0105b40700 1 -- 172.21.15.145:0/2240509801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0100105ea0 msgr2=0x7f010019ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:26:46.040 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f010533f700 1 --2- 172.21.15.145:0/2240509801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0100103580 0x7f0100195c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:26:46.040 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0105b40700 1 --2- 172.21.15.145:0/2240509801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0100105ea0 0x7f010019ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:46.041 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0105b40700 1 -- 172.21.15.145:0/2240509801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0100103580 msgr2=0x7f0100195c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:26:46.041 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0105b40700 1 --2- 172.21.15.145:0/2240509801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0100103580 0x7f0100195c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:26:46.041 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0105b40700 1 -- 172.21.15.145:0/2240509801 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f8009d00 con 0x7f0100100c60 2023-07-12T15:26:46.041 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f0106341700 1 --2- 172.21.15.145:0/2240509801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0100105ea0 0x7f010019ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-12T15:26:46.042 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.030+0000 7f010533f700 1 --2- 172.21.15.145:0/2240509801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0100103580 0x7f0100195c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-12T15:26:46.042 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.031+0000 7f0105b40700 1 --2- 172.21.15.145:0/2240509801 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0100100c60 0x7f01001956f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f00f000c920 tx=0x7f00f000cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:26:46.042 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.031+0000 7f00f6ffd700 1 -- 172.21.15.145:0/2240509801 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f0007b00 con 0x7f0100100c60 2023-07-12T15:26:46.042 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.031+0000 7f00f6ffd700 1 -- 172.21.15.145:0/2240509801 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f00f0004cc0 con 0x7f0100100c60 2023-07-12T15:26:46.042 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.031+0000 7f0107da4700 1 -- 172.21.15.145:0/2240509801 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f010019d830 con 0x7f0100100c60 2023-07-12T15:26:46.042 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.031+0000 7f00f6ffd700 1 -- 172.21.15.145:0/2240509801 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f00054a0 con 0x7f0100100c60 2023-07-12T15:26:46.043 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.031+0000 7f0107da4700 1 -- 172.21.15.145:0/2240509801 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010019ddb0 con 0x7f0100100c60 2023-07-12T15:26:46.043 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.032+0000 7f0107da4700 1 -- 172.21.15.145:0/2240509801 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01000fc210 con 0x7f0100100c60 2023-07-12T15:26:46.043 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.032+0000 7f00f6ffd700 1 -- 172.21.15.145:0/2240509801 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 15) v1 ==== 89784+0+0 (secure 0 0 0) 0x7f00f0004750 con 0x7f0100100c60 2023-07-12T15:26:46.043 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.033+0000 7f00f6ffd700 1 --2- 172.21.15.145:0/2240509801 >> v2:172.21.15.69:6800/284433772 conn(0x7f00ec06c720 0x7f00ec06ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:26:46.043 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.033+0000 7f00f6ffd700 1 -- 172.21.15.145:0/2240509801 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 3497+0+0 (secure 0 0 0) 0x7f00f008abc0 con 0x7f0100100c60 2023-07-12T15:26:46.043 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.033+0000 7f010533f700 1 --2- 172.21.15.145:0/2240509801 >> v2:172.21.15.69:6800/284433772 conn(0x7f00ec06c720 0x7f00ec06ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:26:46.043 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.033+0000 7f010533f700 1 --2- 172.21.15.145:0/2240509801 >> v2:172.21.15.69:6800/284433772 conn(0x7f00ec06c720 0x7f00ec06ebd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f00fc007900 tx=0x7f00fc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:26:46.044 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.037+0000 7f00f6ffd700 1 -- 172.21.15.145:0/2240509801 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f00f0055a50 con 0x7f0100100c60 2023-07-12T15:26:46.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:45 smithi145 ceph-mon[95801]: osdmap e40: 6 total, 6 up, 6 in 2023-07-12T15:26:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:45 smithi145 ceph-mon[95801]: pgmap v121: 1 pgs: 1 remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:26:46.232 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:26:46.230+0000 7f0107da4700 1 -- 172.21.15.145:0/2240509801 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f0100060f40 con 0x7f00ec06c720 2023-07-12T15:26:46.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:45 smithi069 ceph-mon[102777]: osdmap e40: 6 total, 6 up, 6 in 2023-07-12T15:26:46.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:45 smithi069 ceph-mon[102777]: pgmap v121: 1 pgs: 1 remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:26:46.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:45 smithi069 ceph-mon[94241]: osdmap e40: 6 total, 6 up, 6 in 2023-07-12T15:26:46.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:45 smithi069 ceph-mon[94241]: pgmap v121: 1 pgs: 1 remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:26:47.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:46 smithi069 ceph-mon[94241]: osdmap e41: 6 total, 6 up, 6 in 2023-07-12T15:26:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:46 smithi069 ceph-mon[94241]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:26:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:46 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:26:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:46 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:26:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:46 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:47.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:46 smithi069 ceph-mon[102777]: osdmap e41: 6 total, 6 up, 6 in 2023-07-12T15:26:47.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:46 smithi069 ceph-mon[102777]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:26:47.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:46 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:26:47.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:46 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:26:47.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:46 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:47.269 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:46 smithi145 ceph-mon[95801]: osdmap e41: 6 total, 6 up, 6 in 2023-07-12T15:26:47.269 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:46 smithi145 ceph-mon[95801]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:26:47.269 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:46 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:26:47.269 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:46 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:26:47.270 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:46 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:48.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:47 smithi069 ceph-mon[94241]: osdmap e42: 6 total, 6 up, 6 in 2023-07-12T15:26:48.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:47 smithi069 ceph-mon[94241]: pgmap v124: 1 pgs: 1 remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:26:48.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:47 smithi069 ceph-mon[102777]: osdmap e42: 6 total, 6 up, 6 in 2023-07-12T15:26:48.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:47 smithi069 ceph-mon[102777]: pgmap v124: 1 pgs: 1 remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:26:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:47 smithi145 ceph-mon[95801]: osdmap e42: 6 total, 6 up, 6 in 2023-07-12T15:26:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:47 smithi145 ceph-mon[95801]: pgmap v124: 1 pgs: 1 remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:26:49.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/4103432823' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa56dbc-dd4c-487c-8f74-00b05022b978"}]: dispatch 2023-07-12T15:26:49.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa56dbc-dd4c-487c-8f74-00b05022b978"}]: dispatch 2023-07-12T15:26:49.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa56dbc-dd4c-487c-8f74-00b05022b978"}]': finished 2023-07-12T15:26:49.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:48 smithi069 ceph-mon[94241]: osdmap e43: 7 total, 6 up, 7 in 2023-07-12T15:26:49.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:26:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/4103432823' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa56dbc-dd4c-487c-8f74-00b05022b978"}]: dispatch 2023-07-12T15:26:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa56dbc-dd4c-487c-8f74-00b05022b978"}]: dispatch 2023-07-12T15:26:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa56dbc-dd4c-487c-8f74-00b05022b978"}]': finished 2023-07-12T15:26:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:48 smithi069 ceph-mon[102777]: osdmap e43: 7 total, 6 up, 7 in 2023-07-12T15:26:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:26:49.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/4103432823' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa56dbc-dd4c-487c-8f74-00b05022b978"}]: dispatch 2023-07-12T15:26:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa56dbc-dd4c-487c-8f74-00b05022b978"}]: dispatch 2023-07-12T15:26:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa56dbc-dd4c-487c-8f74-00b05022b978"}]': finished 2023-07-12T15:26:49.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:48 smithi145 ceph-mon[95801]: osdmap e43: 7 total, 6 up, 7 in 2023-07-12T15:26:49.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:26:50.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/3380796318' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:26:50.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:49 smithi145 ceph-mon[95801]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 128 KiB/s, 0 objects/s recovering 2023-07-12T15:26:50.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/3380796318' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:26:50.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:49 smithi069 ceph-mon[94241]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 128 KiB/s, 0 objects/s recovering 2023-07-12T15:26:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/3380796318' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:26:50.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:49 smithi069 ceph-mon[102777]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 128 KiB/s, 0 objects/s recovering 2023-07-12T15:26:52.776 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:52 smithi145 ceph-mon[95801]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-12T15:26:52.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:52 smithi069 ceph-mon[94241]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-12T15:26:52.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:52 smithi069 ceph-mon[102777]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-12T15:26:54.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:54 smithi145 ceph-mon[95801]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-12T15:26:54.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:54 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-12T15:26:54.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:54 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:54.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:54 smithi069 ceph-mon[94241]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-12T15:26:54.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-12T15:26:54.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:54.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:54 smithi069 ceph-mon[102777]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-12T15:26:54.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:54 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-12T15:26:54.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:54 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:26:55.615 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:55 smithi145 ceph-mon[95801]: Deploying daemon osd.6 on smithi145 2023-07-12T15:26:55.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:55 smithi069 ceph-mon[94241]: Deploying daemon osd.6 on smithi145 2023-07-12T15:26:55.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:55 smithi069 ceph-mon[102777]: Deploying daemon osd.6 on smithi145 2023-07-12T15:26:56.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:56 smithi145 ceph-mon[95801]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-12T15:26:56.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:56 smithi069 ceph-mon[102777]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-12T15:26:56.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:56 smithi069 ceph-mon[94241]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-12T15:26:58.704 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:26:58 smithi145 ceph-mon[95801]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-12T15:26:58.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:26:58 smithi069 ceph-mon[94241]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-12T15:26:58.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:26:58 smithi069 ceph-mon[102777]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-12T15:27:00.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:00 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:00.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:00 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:00.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:00 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:00.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:00 smithi145 ceph-mon[95801]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:27:00.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:00 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:00.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:00 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:00.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:00 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:00.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:00 smithi069 ceph-mon[94241]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:27:00.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:00 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:00.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:00 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:00.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:00 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:00.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:00 smithi069 ceph-mon[102777]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:27:01.895 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 6 on host 'smithi145' 2023-07-12T15:27:01.895 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:01.892+0000 7f00f6ffd700 1 -- 172.21.15.145:0/2240509801 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0100060f40 con 0x7f00ec06c720 2023-07-12T15:27:01.897 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:01.895+0000 7f0107da4700 1 -- 172.21.15.145:0/2240509801 >> v2:172.21.15.69:6800/284433772 conn(0x7f00ec06c720 msgr2=0x7f00ec06ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:01.898 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:01.895+0000 7f0107da4700 1 --2- 172.21.15.145:0/2240509801 >> v2:172.21.15.69:6800/284433772 conn(0x7f00ec06c720 0x7f00ec06ebd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f00fc007900 tx=0x7f00fc008040 comp rx=0 tx=0).stop 2023-07-12T15:27:01.898 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:01.895+0000 7f0107da4700 1 -- 172.21.15.145:0/2240509801 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0100100c60 msgr2=0x7f01001956f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:01.898 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:01.895+0000 7f0107da4700 1 --2- 172.21.15.145:0/2240509801 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0100100c60 0x7f01001956f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f00f000c920 tx=0x7f00f000cce0 comp rx=0 tx=0).stop 2023-07-12T15:27:01.899 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:01.895+0000 7f0107da4700 1 -- 172.21.15.145:0/2240509801 shutdown_connections 2023-07-12T15:27:01.899 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:01.895+0000 7f0107da4700 1 --2- 172.21.15.145:0/2240509801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0100105ea0 0x7f010019ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:01.899 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:01.895+0000 7f0107da4700 1 --2- 172.21.15.145:0/2240509801 >> v2:172.21.15.69:6800/284433772 conn(0x7f00ec06c720 0x7f00ec06ebd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:01.899 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:01.895+0000 7f0107da4700 1 --2- 172.21.15.145:0/2240509801 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0100100c60 0x7f01001956f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:01.900 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:01.895+0000 7f0107da4700 1 --2- 172.21.15.145:0/2240509801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0100103580 0x7f0100195c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:01.900 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:01.895+0000 7f0107da4700 1 -- 172.21.15.145:0/2240509801 >> 172.21.15.145:0/2240509801 conn(0x7f01000fa6a0 msgr2=0x7f0100101940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:01.900 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:01.896+0000 7f0107da4700 1 -- 172.21.15.145:0/2240509801 shutdown_connections 2023-07-12T15:27:01.900 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:01.896+0000 7f0107da4700 1 -- 172.21.15.145:0/2240509801 wait complete. 2023-07-12T15:27:02.271 DEBUG:teuthology.orchestra.run.smithi145:osd.6> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.6.service 2023-07-12T15:27:02.275 INFO:tasks.cephadm:Deploying osd.7 on smithi145 with /dev/vg_nvme/lv_1... 2023-07-12T15:27:02.275 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-12T15:27:02.306 INFO:journalctl@ceph.osd.6.smithi145.stdout:-- Logs begin at Wed 2023-07-12 15:04:57 UTC. -- 2023-07-12T15:27:02.833 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:02 smithi145 ceph-mon[95801]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:27:02.833 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:02 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.834 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:02 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.834 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:02 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:02.834 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:02 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:02.834 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:02 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.834 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:02 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.834 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:02 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.834 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:02 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:02.835 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:02 smithi145 ceph-mon[95801]: from='osd.6 v2:172.21.15.145:6808/3456495522' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-12T15:27:02.835 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:02 smithi145 ceph-mon[95801]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-12T15:27:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[94241]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:27:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:02.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:02.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[94241]: from='osd.6 v2:172.21.15.145:6808/3456495522' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-12T15:27:02.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[94241]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-12T15:27:02.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[102777]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:27:02.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:02.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:02.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:02.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:02.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[102777]: from='osd.6 v2:172.21.15.145:6808/3456495522' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-12T15:27:02.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:02 smithi069 ceph-mon[102777]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-12T15:27:03.053 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:27:04.100 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:27:03 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:27:03.851+0000 7f3a0430b700 -1 osd.6 0 waiting for initial osdmap 2023-07-12T15:27:04.101 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:27:03 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:27:03.858+0000 7f39fe0fc700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-12T15:27:04.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:03 smithi145 ceph-mon[95801]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-12T15:27:04.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:03 smithi145 ceph-mon[95801]: osdmap e44: 7 total, 6 up, 7 in 2023-07-12T15:27:04.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:03 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:27:04.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:03 smithi145 ceph-mon[95801]: from='osd.6 v2:172.21.15.145:6808/3456495522' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:27:04.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:03 smithi145 ceph-mon[95801]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:27:04.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:03 smithi069 ceph-mon[94241]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-12T15:27:04.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:03 smithi069 ceph-mon[94241]: osdmap e44: 7 total, 6 up, 7 in 2023-07-12T15:27:04.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:03 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:27:04.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:03 smithi069 ceph-mon[94241]: from='osd.6 v2:172.21.15.145:6808/3456495522' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:27:04.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:03 smithi069 ceph-mon[94241]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:27:04.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:03 smithi069 ceph-mon[102777]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-12T15:27:04.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:03 smithi069 ceph-mon[102777]: osdmap e44: 7 total, 6 up, 7 in 2023-07-12T15:27:04.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:03 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:27:04.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:03 smithi069 ceph-mon[102777]: from='osd.6 v2:172.21.15.145:6808/3456495522' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:27:04.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:03 smithi069 ceph-mon[102777]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:27:05.040 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-12T15:27:05.040 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-12T15:27:05.040 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2023-07-12T15:27:05.041 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2023-07-12T15:27:05.041 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111598 s, 940 MB/s 2023-07-12T15:27:05.041 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2023-07-12T15:27:05.179 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:04 smithi145 ceph-mon[95801]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:27:05.179 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:04 smithi145 ceph-mon[95801]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2023-07-12T15:27:05.179 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:04 smithi145 ceph-mon[95801]: osdmap e45: 7 total, 6 up, 7 in 2023-07-12T15:27:05.179 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:04 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:27:05.179 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:04 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:27:05.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:04 smithi069 ceph-mon[102777]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:27:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:04 smithi069 ceph-mon[102777]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2023-07-12T15:27:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:04 smithi069 ceph-mon[102777]: osdmap e45: 7 total, 6 up, 7 in 2023-07-12T15:27:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:04 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:27:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:04 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:27:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:04 smithi069 ceph-mon[94241]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:27:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:04 smithi069 ceph-mon[94241]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2023-07-12T15:27:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:04 smithi069 ceph-mon[94241]: osdmap e45: 7 total, 6 up, 7 in 2023-07-12T15:27:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:04 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:27:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:04 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:27:05.543 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi145:vg_nvme/lv_1 2023-07-12T15:27:06.091 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:27:06.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: purged_snaps scrub starts 2023-07-12T15:27:06.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: purged_snaps scrub ok 2023-07-12T15:27:06.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: osd.6 v2:172.21.15.145:6808/3456495522 boot 2023-07-12T15:27:06.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: osdmap e46: 7 total, 7 up, 7 in 2023-07-12T15:27:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:27:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: osdmap e47: 7 total, 7 up, 7 in 2023-07-12T15:27:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:06.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:06.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:06.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:06.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:06.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: purged_snaps scrub starts 2023-07-12T15:27:06.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: purged_snaps scrub ok 2023-07-12T15:27:06.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: osd.6 v2:172.21.15.145:6808/3456495522 boot 2023-07-12T15:27:06.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: osdmap e46: 7 total, 7 up, 7 in 2023-07-12T15:27:06.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:27:06.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: osdmap e47: 7 total, 7 up, 7 in 2023-07-12T15:27:06.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:06.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:06.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:06.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:06.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:06.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:06.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:06.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:06.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:06.307 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: purged_snaps scrub starts 2023-07-12T15:27:06.307 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: purged_snaps scrub ok 2023-07-12T15:27:06.307 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: osd.6 v2:172.21.15.145:6808/3456495522 boot 2023-07-12T15:27:06.307 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: osdmap e46: 7 total, 7 up, 7 in 2023-07-12T15:27:06.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:27:06.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: osdmap e47: 7 total, 7 up, 7 in 2023-07-12T15:27:06.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:06.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:06.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:06.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:06.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:06.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:06.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:06.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:06.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:06.902 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:06 smithi145 ceph-mon[95801]: Detected new or changed devices on smithi145 2023-07-12T15:27:06.903 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:06 smithi145 ceph-mon[95801]: Adjusting osd_memory_target on smithi145 to 5722M 2023-07-12T15:27:06.903 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:06 smithi145 ceph-mon[95801]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:27:06.903 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:06 smithi145 ceph-mon[95801]: osdmap e48: 7 total, 7 up, 7 in 2023-07-12T15:27:07.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:06 smithi069 ceph-mon[94241]: Detected new or changed devices on smithi145 2023-07-12T15:27:07.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:06 smithi069 ceph-mon[94241]: Adjusting osd_memory_target on smithi145 to 5722M 2023-07-12T15:27:07.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:06 smithi069 ceph-mon[94241]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:27:07.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:06 smithi069 ceph-mon[94241]: osdmap e48: 7 total, 7 up, 7 in 2023-07-12T15:27:07.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:06 smithi069 ceph-mon[102777]: Detected new or changed devices on smithi145 2023-07-12T15:27:07.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:06 smithi069 ceph-mon[102777]: Adjusting osd_memory_target on smithi145 to 5722M 2023-07-12T15:27:07.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:06 smithi069 ceph-mon[102777]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-12T15:27:07.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:06 smithi069 ceph-mon[102777]: osdmap e48: 7 total, 7 up, 7 in 2023-07-12T15:27:07.455 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.453+0000 7f0faa765700 1 -- 172.21.15.145:0/1446201813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fa40fe9f0 msgr2=0x7f0fa40fee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:07.455 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.453+0000 7f0faa765700 1 --2- 172.21.15.145:0/1446201813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fa40fe9f0 0x7f0fa40fee00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009990 tx=0x7f0f8c009ca0 comp rx=0 tx=0).stop 2023-07-12T15:27:07.456 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.453+0000 7f0faa765700 1 -- 172.21.15.145:0/1446201813 shutdown_connections 2023-07-12T15:27:07.456 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.453+0000 7f0faa765700 1 --2- 172.21.15.145:0/1446201813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa40ffcf0 0x7f0fa4100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:07.456 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.453+0000 7f0faa765700 1 --2- 172.21.15.145:0/1446201813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fa40fe9f0 0x7f0fa40fee00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:07.457 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.453+0000 7f0faa765700 1 --2- 172.21.15.145:0/1446201813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fa40ff340 0x7f0fa40ff7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:07.457 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.453+0000 7f0faa765700 1 -- 172.21.15.145:0/1446201813 >> 172.21.15.145:0/1446201813 conn(0x7f0fa40fa6a0 msgr2=0x7f0fa40fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:07.457 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.453+0000 7f0faa765700 1 -- 172.21.15.145:0/1446201813 shutdown_connections 2023-07-12T15:27:07.458 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.453+0000 7f0faa765700 1 -- 172.21.15.145:0/1446201813 wait complete. 2023-07-12T15:27:07.458 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.453+0000 7f0faa765700 1 Processor -- start 2023-07-12T15:27:07.458 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0faa765700 1 -- start start 2023-07-12T15:27:07.460 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0faa765700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa40fe9f0 0x7f0fa41978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:07.460 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0faa765700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fa40ff340 0x7f0fa4197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:07.460 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0faa765700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fa40ffcf0 0x7f0fa419ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:07.460 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0faa765700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa419f450 con 0x7f0fa40fe9f0 2023-07-12T15:27:07.461 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0faa765700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa419f590 con 0x7f0fa40ffcf0 2023-07-12T15:27:07.461 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0faa765700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa419f720 con 0x7f0fa40ff340 2023-07-12T15:27:07.461 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0fa8d02700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fa40ffcf0 0x7f0fa419ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:07.462 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0fa8d02700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fa40ffcf0 0x7f0fa419ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44102/0 (socket says 172.21.15.145:44102) 2023-07-12T15:27:07.462 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0fa8d02700 1 -- 172.21.15.145:0/3348458842 learned_addr learned my addr 172.21.15.145:0/3348458842 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:27:07.462 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0fa3fff700 1 --2- 172.21.15.145:0/3348458842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa40fe9f0 0x7f0fa41978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:07.462 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0fa37fe700 1 --2- 172.21.15.145:0/3348458842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fa40ff340 0x7f0fa4197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:07.463 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0fa8d02700 1 -- 172.21.15.145:0/3348458842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fa40ff340 msgr2=0x7f0fa4197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:07.463 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0fa8d02700 1 --2- 172.21.15.145:0/3348458842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fa40ff340 0x7f0fa4197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:07.463 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0fa8d02700 1 -- 172.21.15.145:0/3348458842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa40fe9f0 msgr2=0x7f0fa41978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:07.464 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0fa8d02700 1 --2- 172.21.15.145:0/3348458842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa40fe9f0 0x7f0fa41978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:07.464 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0fa8d02700 1 -- 172.21.15.145:0/3348458842 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c0095f0 con 0x7f0fa40ffcf0 2023-07-12T15:27:07.464 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.454+0000 7f0fa8d02700 1 --2- 172.21.15.145:0/3348458842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fa40ffcf0 0x7f0fa419ee80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c00b970 tx=0x7f0f9c00bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:07.465 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.455+0000 7f0fa17fa700 1 -- 172.21.15.145:0/3348458842 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c00d860 con 0x7f0fa40ffcf0 2023-07-12T15:27:07.465 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.455+0000 7f0fa17fa700 1 -- 172.21.15.145:0/3348458842 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f0f9c00de50 con 0x7f0fa40ffcf0 2023-07-12T15:27:07.465 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.455+0000 7f0faa765700 1 -- 172.21.15.145:0/3348458842 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa419fa00 con 0x7f0fa40ffcf0 2023-07-12T15:27:07.465 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.455+0000 7f0fa17fa700 1 -- 172.21.15.145:0/3348458842 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c017400 con 0x7f0fa40ffcf0 2023-07-12T15:27:07.466 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.455+0000 7f0faa765700 1 -- 172.21.15.145:0/3348458842 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa419ff30 con 0x7f0fa40ffcf0 2023-07-12T15:27:07.466 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.456+0000 7f0faa765700 1 -- 172.21.15.145:0/3348458842 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa41097d0 con 0x7f0fa40ffcf0 2023-07-12T15:27:07.466 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.456+0000 7f0fa17fa700 1 -- 172.21.15.145:0/3348458842 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 15) v1 ==== 89784+0+0 (secure 0 0 0) 0x7f0f9c00d9c0 con 0x7f0fa40ffcf0 2023-07-12T15:27:07.467 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.456+0000 7f0fa17fa700 1 --2- 172.21.15.145:0/3348458842 >> v2:172.21.15.69:6800/284433772 conn(0x7f0f9006c720 0x7f0f9006ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:07.467 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.457+0000 7f0fa17fa700 1 -- 172.21.15.145:0/3348458842 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 3789+0+0 (secure 0 0 0) 0x7f0f9c006e40 con 0x7f0fa40ffcf0 2023-07-12T15:27:07.467 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.457+0000 7f0fa3fff700 1 --2- 172.21.15.145:0/3348458842 >> v2:172.21.15.69:6800/284433772 conn(0x7f0f9006c720 0x7f0f9006ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:07.467 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.457+0000 7f0fa3fff700 1 --2- 172.21.15.145:0/3348458842 >> v2:172.21.15.69:6800/284433772 conn(0x7f0f9006c720 0x7f0f9006ebd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c00b5c0 tx=0x7f0f8c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:07.468 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.461+0000 7f0fa17fa700 1 -- 172.21.15.145:0/3348458842 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0f9c055a70 con 0x7f0fa40ffcf0 2023-07-12T15:27:07.664 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:07.656+0000 7f0faa765700 1 -- 172.21.15.145:0/3348458842 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fa4060f40 con 0x7f0f9006c720 2023-07-12T15:27:08.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:08 smithi069 ceph-mon[94241]: osdmap e49: 7 total, 7 up, 7 in 2023-07-12T15:27:08.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:08 smithi069 ceph-mon[94241]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-12T15:27:08.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:08 smithi069 ceph-mon[94241]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:08.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:08 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:27:08.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:08 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:27:08.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:08 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:08.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:08 smithi069 ceph-mon[102777]: osdmap e49: 7 total, 7 up, 7 in 2023-07-12T15:27:08.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:08 smithi069 ceph-mon[102777]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-12T15:27:08.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:08 smithi069 ceph-mon[102777]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:08.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:08 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:27:08.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:08 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:27:08.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:08 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:08.497 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:08 smithi145 ceph-mon[95801]: osdmap e49: 7 total, 7 up, 7 in 2023-07-12T15:27:08.498 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:08 smithi145 ceph-mon[95801]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-12T15:27:08.498 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:08 smithi145 ceph-mon[95801]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:08.498 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:08 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-12T15:27:08.498 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:08 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-12T15:27:08.498 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:08 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:09.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:09 smithi069 ceph-mon[94241]: osdmap e50: 7 total, 7 up, 7 in 2023-07-12T15:27:09.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:09 smithi069 ceph-mon[102777]: osdmap e50: 7 total, 7 up, 7 in 2023-07-12T15:27:09.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:09 smithi145 ceph-mon[95801]: osdmap e50: 7 total, 7 up, 7 in 2023-07-12T15:27:10.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:10 smithi069 ceph-mon[102777]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-12T15:27:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/3953340803' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55f492ca-7e77-4643-a798-b0899a6bf944"}]: dispatch 2023-07-12T15:27:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:10 smithi069 ceph-mon[102777]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55f492ca-7e77-4643-a798-b0899a6bf944"}]: dispatch 2023-07-12T15:27:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:10 smithi069 ceph-mon[102777]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55f492ca-7e77-4643-a798-b0899a6bf944"}]': finished 2023-07-12T15:27:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:10 smithi069 ceph-mon[102777]: osdmap e51: 8 total, 7 up, 8 in 2023-07-12T15:27:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:10 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:10 smithi069 ceph-mon[94241]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-12T15:27:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/3953340803' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55f492ca-7e77-4643-a798-b0899a6bf944"}]: dispatch 2023-07-12T15:27:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:10 smithi069 ceph-mon[94241]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55f492ca-7e77-4643-a798-b0899a6bf944"}]: dispatch 2023-07-12T15:27:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:10 smithi069 ceph-mon[94241]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55f492ca-7e77-4643-a798-b0899a6bf944"}]': finished 2023-07-12T15:27:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:10 smithi069 ceph-mon[94241]: osdmap e51: 8 total, 7 up, 8 in 2023-07-12T15:27:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:10 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:10.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:10 smithi145 ceph-mon[95801]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-12T15:27:10.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/3953340803' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55f492ca-7e77-4643-a798-b0899a6bf944"}]: dispatch 2023-07-12T15:27:10.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:10 smithi145 ceph-mon[95801]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55f492ca-7e77-4643-a798-b0899a6bf944"}]: dispatch 2023-07-12T15:27:10.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:10 smithi145 ceph-mon[95801]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55f492ca-7e77-4643-a798-b0899a6bf944"}]': finished 2023-07-12T15:27:10.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:10 smithi145 ceph-mon[95801]: osdmap e51: 8 total, 7 up, 8 in 2023-07-12T15:27:10.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:10 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:11.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/107790746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:27:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/107790746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:27:11.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/107790746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-12T15:27:12.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:12 smithi145 ceph-mon[95801]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-07-12T15:27:12.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:12 smithi069 ceph-mon[94241]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-07-12T15:27:12.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:12 smithi069 ceph-mon[102777]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-07-12T15:27:14.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:14 smithi145 ceph-mon[95801]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2023-07-12T15:27:14.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:14 smithi069 ceph-mon[94241]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2023-07-12T15:27:14.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:14 smithi069 ceph-mon[102777]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2023-07-12T15:27:16.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:15 smithi069 ceph-mon[94241]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-12T15:27:16.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:15 smithi069 ceph-mon[102777]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-12T15:27:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:15 smithi145 ceph-mon[95801]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-12T15:27:17.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:16 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-12T15:27:17.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:16 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:17.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:16 smithi069 ceph-mon[94241]: Deploying daemon osd.7 on smithi145 2023-07-12T15:27:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:16 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-12T15:27:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:16 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:16 smithi069 ceph-mon[102777]: Deploying daemon osd.7 on smithi145 2023-07-12T15:27:17.288 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:16 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-12T15:27:17.289 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:16 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:17.289 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:16 smithi145 ceph-mon[95801]: Deploying daemon osd.7 on smithi145 2023-07-12T15:27:18.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:17 smithi069 ceph-mon[94241]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-07-12T15:27:18.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:17 smithi069 ceph-mon[102777]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-07-12T15:27:18.347 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:17 smithi145 ceph-mon[95801]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-07-12T15:27:20.599 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:20 smithi145 ceph-mon[95801]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-12T15:27:20.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:20 smithi069 ceph-mon[94241]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-12T15:27:20.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:20 smithi069 ceph-mon[102777]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-12T15:27:21.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:21 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:21.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:21 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:21.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:21 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:21.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:21 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:21 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:21 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:21.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:21.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:21.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:21 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:22.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:22 smithi069 ceph-mon[94241]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-07-12T15:27:22.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:22 smithi069 ceph-mon[102777]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-07-12T15:27:23.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:22 smithi145 ceph-mon[95801]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-07-12T15:27:23.563 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 7 on host 'smithi145' 2023-07-12T15:27:23.564 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:23.561+0000 7f0fa17fa700 1 -- 172.21.15.145:0/3348458842 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0fa4060f40 con 0x7f0f9006c720 2023-07-12T15:27:23.565 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:23.564+0000 7f0faa765700 1 -- 172.21.15.145:0/3348458842 >> v2:172.21.15.69:6800/284433772 conn(0x7f0f9006c720 msgr2=0x7f0f9006ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:23.566 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:23.564+0000 7f0faa765700 1 --2- 172.21.15.145:0/3348458842 >> v2:172.21.15.69:6800/284433772 conn(0x7f0f9006c720 0x7f0f9006ebd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c00b5c0 tx=0x7f0f8c005fd0 comp rx=0 tx=0).stop 2023-07-12T15:27:23.566 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:23.564+0000 7f0faa765700 1 -- 172.21.15.145:0/3348458842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fa40ffcf0 msgr2=0x7f0fa419ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:23.566 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:23.564+0000 7f0faa765700 1 --2- 172.21.15.145:0/3348458842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fa40ffcf0 0x7f0fa419ee80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c00b970 tx=0x7f0f9c00bc80 comp rx=0 tx=0).stop 2023-07-12T15:27:23.567 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:23.564+0000 7f0faa765700 1 -- 172.21.15.145:0/3348458842 shutdown_connections 2023-07-12T15:27:23.567 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:23.564+0000 7f0faa765700 1 --2- 172.21.15.145:0/3348458842 >> v2:172.21.15.69:6800/284433772 conn(0x7f0f9006c720 0x7f0f9006ebd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:23.567 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:23.564+0000 7f0faa765700 1 --2- 172.21.15.145:0/3348458842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fa40ffcf0 0x7f0fa419ee80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:23.568 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:23.564+0000 7f0faa765700 1 --2- 172.21.15.145:0/3348458842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa40fe9f0 0x7f0fa41978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:23.568 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:23.564+0000 7f0faa765700 1 --2- 172.21.15.145:0/3348458842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fa40ff340 0x7f0fa4197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:23.568 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:23.564+0000 7f0faa765700 1 -- 172.21.15.145:0/3348458842 >> 172.21.15.145:0/3348458842 conn(0x7f0fa40fa6a0 msgr2=0x7f0fa40fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:23.568 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:23.564+0000 7f0faa765700 1 -- 172.21.15.145:0/3348458842 shutdown_connections 2023-07-12T15:27:23.569 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:23.564+0000 7f0faa765700 1 -- 172.21.15.145:0/3348458842 wait complete. 2023-07-12T15:27:23.985 DEBUG:teuthology.orchestra.run.smithi145:osd.7> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.7.service 2023-07-12T15:27:23.987 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2023-07-12T15:27:23.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-12T15:27:24.017 INFO:journalctl@ceph.osd.7.smithi145.stdout:-- Logs begin at Wed 2023-07-12 15:04:57 UTC. -- 2023-07-12T15:27:24.553 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:27:24.587 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:24 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:24.588 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:24 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:24.588 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:24 smithi145 ceph-mon[95801]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-12T15:27:24.588 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:24 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:24.588 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:24 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:24.588 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:24 smithi145 ceph-mon[95801]: from='osd.7 v2:172.21.15.145:6812/2623299546' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-12T15:27:24.589 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:24 smithi145 ceph-mon[95801]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-12T15:27:24.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:24.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:24.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[94241]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-12T15:27:24.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:24.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:24.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[94241]: from='osd.7 v2:172.21.15.145:6812/2623299546' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-12T15:27:24.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[94241]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-12T15:27:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[102777]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-12T15:27:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[102777]: from='osd.7 v2:172.21.15.145:6812/2623299546' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-12T15:27:24.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:24 smithi069 ceph-mon[102777]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-12T15:27:25.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:25 smithi145 ceph-mon[95801]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-12T15:27:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:25 smithi145 ceph-mon[95801]: osdmap e52: 8 total, 7 up, 8 in 2023-07-12T15:27:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:25 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:25 smithi145 ceph-mon[95801]: from='osd.7 v2:172.21.15.145:6812/2623299546' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:27:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:25 smithi145 ceph-mon[95801]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:27:25.602 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:27:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:27:25.547+0000 7f6eb8c0c700 -1 osd.7 0 waiting for initial osdmap 2023-07-12T15:27:25.602 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:27:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:27:25.556+0000 7f6eb39ff700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-12T15:27:25.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.887+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/678689036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b08101df0 msgr2=0x7f2b081086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:25.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.887+0000 7f2b0d8fd700 1 --2- 172.21.15.69:0/678689036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b08101df0 0x7f2b081086b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009f90 tx=0x7f2b0000b330 comp rx=0 tx=0).stop 2023-07-12T15:27:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.887+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/678689036 shutdown_connections 2023-07-12T15:27:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.887+0000 7f2b0d8fd700 1 --2- 172.21.15.69:0/678689036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b08101df0 0x7f2b081086b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.887+0000 7f2b0d8fd700 1 --2- 172.21.15.69:0/678689036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b080fe480 0x7f2b080fe890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.887+0000 7f2b0d8fd700 1 --2- 172.21.15.69:0/678689036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b08101460 0x7f2b081018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.887+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/678689036 >> 172.21.15.69:0/678689036 conn(0x7f2b080f9e70 msgr2=0x7f2b080fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.887+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/678689036 shutdown_connections 2023-07-12T15:27:25.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.887+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/678689036 wait complete. 2023-07-12T15:27:25.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.888+0000 7f2b0d8fd700 1 Processor -- start 2023-07-12T15:27:25.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.888+0000 7f2b0d8fd700 1 -- start start 2023-07-12T15:27:25.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.888+0000 7f2b0d8fd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b080fe480 0x7f2b08195680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:25.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.888+0000 7f2b0d8fd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b08101460 0x7f2b08195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:25.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.888+0000 7f2b0d8fd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b08101df0 0x7f2b0819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:25.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.888+0000 7f2b0d8fd700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0819d210 con 0x7f2b08101460 2023-07-12T15:27:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.888+0000 7f2b0d8fd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0819d350 con 0x7f2b080fe480 2023-07-12T15:27:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.888+0000 7f2b0d8fd700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0819d4e0 con 0x7f2b08101df0 2023-07-12T15:27:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.888+0000 7f2b067fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b08101460 0x7f2b08195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.888+0000 7f2b077fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b08101df0 0x7f2b0819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.888+0000 7f2b067fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b08101460 0x7f2b08195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40356/0 (socket says 172.21.15.69:40356) 2023-07-12T15:27:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.888+0000 7f2b077fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b08101df0 0x7f2b0819cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:33786/0 (socket says 172.21.15.69:33786) 2023-07-12T15:27:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.888+0000 7f2b067fc700 1 -- 172.21.15.69:0/441964722 learned_addr learned my addr 172.21.15.69:0/441964722 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:27:25.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b06ffd700 1 --2- 172.21.15.69:0/441964722 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b080fe480 0x7f2b08195680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:25.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b067fc700 1 -- 172.21.15.69:0/441964722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b08101df0 msgr2=0x7f2b0819cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:25.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b067fc700 1 --2- 172.21.15.69:0/441964722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b08101df0 0x7f2b0819cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:25.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b067fc700 1 -- 172.21.15.69:0/441964722 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b080fe480 msgr2=0x7f2b08195680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:25.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b067fc700 1 --2- 172.21.15.69:0/441964722 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b080fe480 0x7f2b08195680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:25.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b067fc700 1 -- 172.21.15.69:0/441964722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af80095f0 con 0x7f2b08101460 2023-07-12T15:27:25.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b077fe700 1 --2- 172.21.15.69:0/441964722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b08101df0 0x7f2b0819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:27:25.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b06ffd700 1 --2- 172.21.15.69:0/441964722 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b080fe480 0x7f2b08195680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-12T15:27:25.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b067fc700 1 --2- 172.21.15.69:0/441964722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b08101460 0x7f2b08195bc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2af800ec40 tx=0x7f2af800ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:25.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b0c8fb700 1 -- 172.21.15.69:0/441964722 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af8009a10 con 0x7f2b08101460 2023-07-12T15:27:25.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b0c8fb700 1 -- 172.21.15.69:0/441964722 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f2af80044b0 con 0x7f2b08101460 2023-07-12T15:27:25.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/441964722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b00009c40 con 0x7f2b08101460 2023-07-12T15:27:25.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b0c8fb700 1 -- 172.21.15.69:0/441964722 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af8003ea0 con 0x7f2b08101460 2023-07-12T15:27:25.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.889+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/441964722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b0819db30 con 0x7f2b08101460 2023-07-12T15:27:25.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.890+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/441964722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b08107700 con 0x7f2b08101460 2023-07-12T15:27:25.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.890+0000 7f2b0c8fb700 1 -- 172.21.15.69:0/441964722 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 15) v1 ==== 89784+0+0 (secure 0 0 0) 0x7f2af8004000 con 0x7f2b08101460 2023-07-12T15:27:25.896 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.891+0000 7f2b0c8fb700 1 --2- 172.21.15.69:0/441964722 >> v2:172.21.15.69:6800/284433772 conn(0x7f2af406c720 0x7f2af406ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.891+0000 7f2b0c8fb700 1 -- 172.21.15.69:0/441964722 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 3941+0+0 (secure 0 0 0) 0x7f2af8014070 con 0x7f2b08101460 2023-07-12T15:27:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.891+0000 7f2b06ffd700 1 --2- 172.21.15.69:0/441964722 >> v2:172.21.15.69:6800/284433772 conn(0x7f2af406c720 0x7f2af406ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.891+0000 7f2b06ffd700 1 --2- 172.21.15.69:0/441964722 >> v2:172.21.15.69:6800/284433772 conn(0x7f2af406c720 0x7f2af406ebd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b08068d60 tx=0x7f2af0009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:25.895+0000 7f2b0c8fb700 1 -- 172.21.15.69:0/441964722 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2af8056470 con 0x7f2b08101460 2023-07-12T15:27:25.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:25 smithi069 ceph-mon[94241]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-12T15:27:25.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:25 smithi069 ceph-mon[94241]: osdmap e52: 8 total, 7 up, 8 in 2023-07-12T15:27:25.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:25 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:25.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:25 smithi069 ceph-mon[94241]: from='osd.7 v2:172.21.15.145:6812/2623299546' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:27:25.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:25 smithi069 ceph-mon[94241]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:27:25.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:25 smithi069 ceph-mon[102777]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-12T15:27:25.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:25 smithi069 ceph-mon[102777]: osdmap e52: 8 total, 7 up, 8 in 2023-07-12T15:27:25.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:25 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:25.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:25 smithi069 ceph-mon[102777]: from='osd.7 v2:172.21.15.145:6812/2623299546' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:27:25.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:25 smithi069 ceph-mon[102777]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2023-07-12T15:27:26.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.084+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/441964722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f2b0802cfb0 con 0x7f2b08101460 2023-07-12T15:27:26.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.084+0000 7f2b0c8fb700 1 -- 172.21.15.69:0/441964722 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v53) v1 ==== 74+0+130 (secure 0 0 0) 0x7f2af8059a90 con 0x7f2b08101460 2023-07-12T15:27:26.086 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:27:26.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.086+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/441964722 >> v2:172.21.15.69:6800/284433772 conn(0x7f2af406c720 msgr2=0x7f2af406ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:26.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.086+0000 7f2b0d8fd700 1 --2- 172.21.15.69:0/441964722 >> v2:172.21.15.69:6800/284433772 conn(0x7f2af406c720 0x7f2af406ebd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b08068d60 tx=0x7f2af0009670 comp rx=0 tx=0).stop 2023-07-12T15:27:26.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.086+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/441964722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b08101460 msgr2=0x7f2b08195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:26.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.086+0000 7f2b0d8fd700 1 --2- 172.21.15.69:0/441964722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b08101460 0x7f2b08195bc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2af800ec40 tx=0x7f2af800ef50 comp rx=0 tx=0).stop 2023-07-12T15:27:26.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.086+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/441964722 shutdown_connections 2023-07-12T15:27:26.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.086+0000 7f2b0d8fd700 1 --2- 172.21.15.69:0/441964722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b08101df0 0x7f2b0819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:26.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.086+0000 7f2b0d8fd700 1 --2- 172.21.15.69:0/441964722 >> v2:172.21.15.69:6800/284433772 conn(0x7f2af406c720 0x7f2af406ebd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:26.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.086+0000 7f2b0d8fd700 1 --2- 172.21.15.69:0/441964722 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b080fe480 0x7f2b08195680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:26.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.086+0000 7f2b0d8fd700 1 --2- 172.21.15.69:0/441964722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b08101460 0x7f2b08195bc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:26.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.086+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/441964722 >> 172.21.15.69:0/441964722 conn(0x7f2b080f9e70 msgr2=0x7f2b08104840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:26.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.087+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/441964722 shutdown_connections 2023-07-12T15:27:26.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:26.087+0000 7f2b0d8fd700 1 -- 172.21.15.69:0/441964722 wait complete. 2023-07-12T15:27:26.509 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1689175624,"num_in_osds":8,"osd_in_since":1689175630,"num_remapped_pgs":0} 2023-07-12T15:27:26.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-12T15:27:26.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2023-07-12T15:27:26.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: osdmap e53: 8 total, 7 up, 8 in 2023-07-12T15:27:26.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:26.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:26.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/441964722' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-12T15:27:26.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:26.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:26.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:26.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:26.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:26.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:26.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:26.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:26.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:26.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: osd.7 v2:172.21.15.145:6812/2623299546 boot 2023-07-12T15:27:26.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: osdmap e54: 8 total, 8 up, 8 in 2023-07-12T15:27:26.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:26 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:26.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-12T15:27:26.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2023-07-12T15:27:26.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: osdmap e53: 8 total, 7 up, 8 in 2023-07-12T15:27:26.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:26.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:26.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/441964722' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-12T15:27:26.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:26.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:26.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:26.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:26.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:26.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:26.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:26.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:26.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:26.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: osd.7 v2:172.21.15.145:6812/2623299546 boot 2023-07-12T15:27:26.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: osdmap e54: 8 total, 8 up, 8 in 2023-07-12T15:27:26.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:26.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-12T15:27:26.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2023-07-12T15:27:26.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: osdmap e53: 8 total, 7 up, 8 in 2023-07-12T15:27:26.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:26.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:26.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/441964722' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-12T15:27:26.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:26.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:26.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:26.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:27:26.992 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.992 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:26.992 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:26.992 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.992 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:26.992 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:26.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:26.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:26.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: osd.7 v2:172.21.15.145:6812/2623299546 boot 2023-07-12T15:27:26.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: osdmap e54: 8 total, 8 up, 8 in 2023-07-12T15:27:26.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:26 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:27:27.510 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-12T15:27:27.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:27 smithi145 ceph-mon[95801]: purged_snaps scrub starts 2023-07-12T15:27:27.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:27 smithi145 ceph-mon[95801]: purged_snaps scrub ok 2023-07-12T15:27:27.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:27 smithi145 ceph-mon[95801]: Detected new or changed devices on smithi145 2023-07-12T15:27:27.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:27 smithi145 ceph-mon[95801]: Adjusting osd_memory_target on smithi145 to 4292M 2023-07-12T15:27:27.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:27 smithi145 ceph-mon[95801]: osdmap e55: 8 total, 8 up, 8 in 2023-07-12T15:27:27.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:27 smithi069 ceph-mon[102777]: purged_snaps scrub starts 2023-07-12T15:27:27.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:27 smithi069 ceph-mon[102777]: purged_snaps scrub ok 2023-07-12T15:27:27.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:27 smithi069 ceph-mon[102777]: Detected new or changed devices on smithi145 2023-07-12T15:27:27.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:27 smithi069 ceph-mon[102777]: Adjusting osd_memory_target on smithi145 to 4292M 2023-07-12T15:27:27.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:27 smithi069 ceph-mon[102777]: osdmap e55: 8 total, 8 up, 8 in 2023-07-12T15:27:27.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:27 smithi069 ceph-mon[94241]: purged_snaps scrub starts 2023-07-12T15:27:27.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:27 smithi069 ceph-mon[94241]: purged_snaps scrub ok 2023-07-12T15:27:27.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:27 smithi069 ceph-mon[94241]: Detected new or changed devices on smithi145 2023-07-12T15:27:27.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:27 smithi069 ceph-mon[94241]: Adjusting osd_memory_target on smithi145 to 4292M 2023-07-12T15:27:27.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:27 smithi069 ceph-mon[94241]: osdmap e55: 8 total, 8 up, 8 in 2023-07-12T15:27:28.076 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:27:28.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:28 smithi069 ceph-mon[102777]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:27:28.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:28 smithi069 ceph-mon[102777]: osdmap e56: 8 total, 8 up, 8 in 2023-07-12T15:27:28.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:28 smithi069 ceph-mon[94241]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:27:28.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:28 smithi069 ceph-mon[94241]: osdmap e56: 8 total, 8 up, 8 in 2023-07-12T15:27:28.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:28 smithi145 ceph-mon[95801]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:27:28.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:28 smithi145 ceph-mon[95801]: osdmap e56: 8 total, 8 up, 8 in 2023-07-12T15:27:29.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.240+0000 7f99b0867700 1 -- 172.21.15.69:0/2585889667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ac0ffca0 msgr2=0x7f99ac107550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:29.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.240+0000 7f99b0867700 1 --2- 172.21.15.69:0/2585889667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ac0ffca0 0x7f99ac107550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f99a400b0a0 tx=0x7f99a400b3b0 comp rx=0 tx=0).stop 2023-07-12T15:27:29.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.240+0000 7f99b0867700 1 -- 172.21.15.69:0/2585889667 shutdown_connections 2023-07-12T15:27:29.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.240+0000 7f99b0867700 1 --2- 172.21.15.69:0/2585889667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ac0ffca0 0x7f99ac107550 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:29.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.240+0000 7f99b0867700 1 --2- 172.21.15.69:0/2585889667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99ac100690 0x7f99ac100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:29.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.240+0000 7f99b0867700 1 --2- 172.21.15.69:0/2585889667 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99ac0ff2f0 0x7f99ac0ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:29.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.240+0000 7f99b0867700 1 -- 172.21.15.69:0/2585889667 >> 172.21.15.69:0/2585889667 conn(0x7f99ac0f9e50 msgr2=0x7f99ac0fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:29.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.240+0000 7f99b0867700 1 -- 172.21.15.69:0/2585889667 shutdown_connections 2023-07-12T15:27:29.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.240+0000 7f99b0867700 1 -- 172.21.15.69:0/2585889667 wait complete. 2023-07-12T15:27:29.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.241+0000 7f99b0867700 1 Processor -- start 2023-07-12T15:27:29.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.241+0000 7f99b0867700 1 -- start start 2023-07-12T15:27:29.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.241+0000 7f99b0867700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ac0ff2f0 0x7f99ac195680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:29.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.241+0000 7f99b0867700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99ac0ffca0 0x7f99ac195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:29.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.241+0000 7f99b0867700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99ac100690 0x7f99ac19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:29.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.241+0000 7f99b0867700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac19d210 con 0x7f99ac0ff2f0 2023-07-12T15:27:29.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.241+0000 7f99b0867700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac19d350 con 0x7f99ac0ffca0 2023-07-12T15:27:29.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.241+0000 7f99b0867700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac19d4e0 con 0x7f99ac100690 2023-07-12T15:27:29.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.241+0000 7f99aa59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ac0ff2f0 0x7f99ac195680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:29.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.241+0000 7f99aad9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99ac100690 0x7f99ac19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:29.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.241+0000 7f99aa59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ac0ff2f0 0x7f99ac195680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40382/0 (socket says 172.21.15.69:40382) 2023-07-12T15:27:29.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.241+0000 7f99aa59c700 1 -- 172.21.15.69:0/3565386249 learned_addr learned my addr 172.21.15.69:0/3565386249 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:27:29.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f99a9d9b700 1 --2- 172.21.15.69:0/3565386249 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99ac0ffca0 0x7f99ac195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:29.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f99aad9d700 1 -- 172.21.15.69:0/3565386249 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99ac0ffca0 msgr2=0x7f99ac195bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:29.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f99aad9d700 1 --2- 172.21.15.69:0/3565386249 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99ac0ffca0 0x7f99ac195bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:29.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f99aad9d700 1 -- 172.21.15.69:0/3565386249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ac0ff2f0 msgr2=0x7f99ac195680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:29.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f99aad9d700 1 --2- 172.21.15.69:0/3565386249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ac0ff2f0 0x7f99ac195680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:29.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f99aad9d700 1 -- 172.21.15.69:0/3565386249 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c009520 con 0x7f99ac100690 2023-07-12T15:27:29.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f99aa59c700 1 --2- 172.21.15.69:0/3565386249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ac0ff2f0 0x7f99ac195680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:27:29.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f99a9d9b700 1 --2- 172.21.15.69:0/3565386249 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99ac0ffca0 0x7f99ac195bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-12T15:27:29.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f99aad9d700 1 --2- 172.21.15.69:0/3565386249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99ac100690 0x7f99ac19cc40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f99a400b0a0 tx=0x7f99a400bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:29.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f999b7fe700 1 -- 172.21.15.69:0/3565386249 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a400e040 con 0x7f99ac100690 2023-07-12T15:27:29.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f999b7fe700 1 -- 172.21.15.69:0/3565386249 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f99a4004c50 con 0x7f99ac100690 2023-07-12T15:27:29.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f999b7fe700 1 -- 172.21.15.69:0/3565386249 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a40052a0 con 0x7f99ac100690 2023-07-12T15:27:29.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f99b0867700 1 -- 172.21.15.69:0/3565386249 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a4009d00 con 0x7f99ac100690 2023-07-12T15:27:29.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.242+0000 7f99b0867700 1 -- 172.21.15.69:0/3565386249 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ac19d9f0 con 0x7f99ac100690 2023-07-12T15:27:29.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.243+0000 7f99b0867700 1 -- 172.21.15.69:0/3565386249 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ac1066f0 con 0x7f99ac100690 2023-07-12T15:27:29.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.243+0000 7f999b7fe700 1 -- 172.21.15.69:0/3565386249 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 15) v1 ==== 89784+0+0 (secure 0 0 0) 0x7f99a401d470 con 0x7f99ac100690 2023-07-12T15:27:29.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.244+0000 7f999b7fe700 1 --2- 172.21.15.69:0/3565386249 >> v2:172.21.15.69:6800/284433772 conn(0x7f999406c720 0x7f999406ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:29.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.244+0000 7f999b7fe700 1 -- 172.21.15.69:0/3565386249 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4110+0+0 (secure 0 0 0) 0x7f99a408c860 con 0x7f99ac100690 2023-07-12T15:27:29.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.244+0000 7f99aa59c700 1 --2- 172.21.15.69:0/3565386249 >> v2:172.21.15.69:6800/284433772 conn(0x7f999406c720 0x7f999406ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:29.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.244+0000 7f99aa59c700 1 --2- 172.21.15.69:0/3565386249 >> v2:172.21.15.69:6800/284433772 conn(0x7f999406c720 0x7f999406ebd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f99a00105c0 tx=0x7f99a000b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:29.250 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.249+0000 7f999b7fe700 1 -- 172.21.15.69:0/3565386249 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f99a4057510 con 0x7f99ac100690 2023-07-12T15:27:29.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.433+0000 7f99b0867700 1 -- 172.21.15.69:0/3565386249 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f99ac02cb20 con 0x7f99ac100690 2023-07-12T15:27:29.435 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.434+0000 7f999b7fe700 1 -- 172.21.15.69:0/3565386249 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v56) v1 ==== 74+0+130 (secure 0 0 0) 0x7f99ac02cb20 con 0x7f99ac100690 2023-07-12T15:27:29.436 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:27:29.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.436+0000 7f99b0867700 1 -- 172.21.15.69:0/3565386249 >> v2:172.21.15.69:6800/284433772 conn(0x7f999406c720 msgr2=0x7f999406ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:29.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.436+0000 7f99b0867700 1 --2- 172.21.15.69:0/3565386249 >> v2:172.21.15.69:6800/284433772 conn(0x7f999406c720 0x7f999406ebd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f99a00105c0 tx=0x7f99a000b4a0 comp rx=0 tx=0).stop 2023-07-12T15:27:29.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.436+0000 7f99b0867700 1 -- 172.21.15.69:0/3565386249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99ac100690 msgr2=0x7f99ac19cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:29.437 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.436+0000 7f99b0867700 1 --2- 172.21.15.69:0/3565386249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99ac100690 0x7f99ac19cc40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f99a400b0a0 tx=0x7f99a400bab0 comp rx=0 tx=0).stop 2023-07-12T15:27:29.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.436+0000 7f99b0867700 1 -- 172.21.15.69:0/3565386249 shutdown_connections 2023-07-12T15:27:29.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.436+0000 7f99b0867700 1 --2- 172.21.15.69:0/3565386249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99ac100690 0x7f99ac19cc40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:29.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.436+0000 7f99b0867700 1 --2- 172.21.15.69:0/3565386249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ac0ff2f0 0x7f99ac195680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:29.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.436+0000 7f99b0867700 1 --2- 172.21.15.69:0/3565386249 >> v2:172.21.15.69:6800/284433772 conn(0x7f999406c720 0x7f999406ebd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:29.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.436+0000 7f99b0867700 1 --2- 172.21.15.69:0/3565386249 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99ac0ffca0 0x7f99ac195bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:29.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.436+0000 7f99b0867700 1 -- 172.21.15.69:0/3565386249 >> 172.21.15.69:0/3565386249 conn(0x7f99ac0f9e50 msgr2=0x7f99ac103830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:29.438 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.436+0000 7f99b0867700 1 -- 172.21.15.69:0/3565386249 shutdown_connections 2023-07-12T15:27:29.439 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:29.436+0000 7f99b0867700 1 -- 172.21.15.69:0/3565386249 wait complete. 2023-07-12T15:27:29.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3565386249' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-12T15:27:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3565386249' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-12T15:27:29.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3565386249' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-12T15:27:29.918 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1689175646,"num_in_osds":8,"osd_in_since":1689175630,"num_remapped_pgs":1} 2023-07-12T15:27:29.919 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd dump --format=json 2023-07-12T15:27:30.481 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:27:30.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:30 smithi145 ceph-mon[95801]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-12T15:27:30.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:30 smithi145 ceph-mon[95801]: osdmap e57: 8 total, 8 up, 8 in 2023-07-12T15:27:30.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:30 smithi069 ceph-mon[102777]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-12T15:27:30.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:30 smithi069 ceph-mon[102777]: osdmap e57: 8 total, 8 up, 8 in 2023-07-12T15:27:30.859 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:30 smithi069 ceph-mon[94241]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-12T15:27:30.860 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:30 smithi069 ceph-mon[94241]: osdmap e57: 8 total, 8 up, 8 in 2023-07-12T15:27:31.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.632+0000 7f04301a6700 1 -- 172.21.15.69:0/2756320268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04280ffca0 msgr2=0x7f0428103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:31.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.632+0000 7f04301a6700 1 --2- 172.21.15.69:0/2756320268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04280ffca0 0x7f0428103c70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0424009920 tx=0x7f0424009c30 comp rx=0 tx=0).stop 2023-07-12T15:27:31.635 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.632+0000 7f04301a6700 1 -- 172.21.15.69:0/2756320268 shutdown_connections 2023-07-12T15:27:31.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.632+0000 7f04301a6700 1 --2- 172.21.15.69:0/2756320268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04281043a0 0x7f042810bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:31.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.632+0000 7f04301a6700 1 --2- 172.21.15.69:0/2756320268 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04280ff300 0x7f04280ff6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:31.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.632+0000 7f04301a6700 1 --2- 172.21.15.69:0/2756320268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04280ffca0 0x7f0428103c70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:31.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.632+0000 7f04301a6700 1 -- 172.21.15.69:0/2756320268 >> 172.21.15.69:0/2756320268 conn(0x7f0428076160 msgr2=0x7f0428076560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:31.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.632+0000 7f04301a6700 1 -- 172.21.15.69:0/2756320268 shutdown_connections 2023-07-12T15:27:31.636 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.632+0000 7f04301a6700 1 -- 172.21.15.69:0/2756320268 wait complete. 2023-07-12T15:27:31.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f04301a6700 1 Processor -- start 2023-07-12T15:27:31.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f04301a6700 1 -- start start 2023-07-12T15:27:31.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f04301a6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04280ff300 0x7f0428195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:31.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f04301a6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04280ffca0 0x7f0428196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:31.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f04301a6700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04281043a0 0x7f042819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:31.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f04301a6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042819ad10 con 0x7f04280ff300 2023-07-12T15:27:31.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f04301a6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042819ae80 con 0x7f04280ffca0 2023-07-12T15:27:31.637 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f04301a6700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042819b040 con 0x7f04281043a0 2023-07-12T15:27:31.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f042df42700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04280ff300 0x7f0428195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:31.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f042e743700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04281043a0 0x7f042819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:31.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f042e743700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04281043a0 0x7f042819a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:33824/0 (socket says 172.21.15.69:33824) 2023-07-12T15:27:31.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f042df42700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04280ff300 0x7f0428195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40392/0 (socket says 172.21.15.69:40392) 2023-07-12T15:27:31.638 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f042e743700 1 -- 172.21.15.69:0/1376984558 learned_addr learned my addr 172.21.15.69:0/1376984558 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:27:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f042d741700 1 --2- 172.21.15.69:0/1376984558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04280ffca0 0x7f0428196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f042e743700 1 -- 172.21.15.69:0/1376984558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04280ffca0 msgr2=0x7f0428196290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f042e743700 1 --2- 172.21.15.69:0/1376984558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04280ffca0 0x7f0428196290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f042e743700 1 -- 172.21.15.69:0/1376984558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04280ff300 msgr2=0x7f0428195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f042e743700 1 --2- 172.21.15.69:0/1376984558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04280ff300 0x7f0428195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f042e743700 1 -- 172.21.15.69:0/1376984558 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0420009d00 con 0x7f04281043a0 2023-07-12T15:27:31.639 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f042df42700 1 --2- 172.21.15.69:0/1376984558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04280ff300 0x7f0428195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:27:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.633+0000 7f042d741700 1 --2- 172.21.15.69:0/1376984558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04280ffca0 0x7f0428196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:27:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.634+0000 7f042e743700 1 --2- 172.21.15.69:0/1376984558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04281043a0 0x7f042819a700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f042000c340 tx=0x7f042000c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.634+0000 7f041effd700 1 -- 172.21.15.69:0/1376984558 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042000e050 con 0x7f04281043a0 2023-07-12T15:27:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.634+0000 7f041effd700 1 -- 172.21.15.69:0/1376984558 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f042000f040 con 0x7f04281043a0 2023-07-12T15:27:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.634+0000 7f041effd700 1 -- 172.21.15.69:0/1376984558 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04200079b0 con 0x7f04281043a0 2023-07-12T15:27:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.634+0000 7f04301a6700 1 -- 172.21.15.69:0/1376984558 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0424009520 con 0x7f04281043a0 2023-07-12T15:27:31.640 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.634+0000 7f04301a6700 1 -- 172.21.15.69:0/1376984558 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04281a29b0 con 0x7f04281043a0 2023-07-12T15:27:31.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.635+0000 7f04301a6700 1 -- 172.21.15.69:0/1376984558 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042804e800 con 0x7f04281043a0 2023-07-12T15:27:31.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.635+0000 7f041effd700 1 -- 172.21.15.69:0/1376984558 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 15) v1 ==== 89784+0+0 (secure 0 0 0) 0x7f0420007b10 con 0x7f04281043a0 2023-07-12T15:27:31.643 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.636+0000 7f041effd700 1 --2- 172.21.15.69:0/1376984558 >> v2:172.21.15.69:6800/284433772 conn(0x7f041406c720 0x7f041406ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:31.644 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.636+0000 7f041effd700 1 -- 172.21.15.69:0/1376984558 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (secure 0 0 0) 0x7f042008b020 con 0x7f04281043a0 2023-07-12T15:27:31.644 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.636+0000 7f042df42700 1 --2- 172.21.15.69:0/1376984558 >> v2:172.21.15.69:6800/284433772 conn(0x7f041406c720 0x7f041406ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:31.644 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.636+0000 7f042df42700 1 --2- 172.21.15.69:0/1376984558 >> v2:172.21.15.69:6800/284433772 conn(0x7f041406c720 0x7f041406ebd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0418009de0 tx=0x7f0418009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:31.645 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.640+0000 7f041effd700 1 -- 172.21.15.69:0/1376984558 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0420055c60 con 0x7f04281043a0 2023-07-12T15:27:31.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:31 smithi069 ceph-mon[102777]: osdmap e58: 8 total, 8 up, 8 in 2023-07-12T15:27:31.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:31 smithi069 ceph-mon[94241]: osdmap e58: 8 total, 8 up, 8 in 2023-07-12T15:27:31.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.827+0000 7f04301a6700 1 -- 172.21.15.69:0/1376984558 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04281a2b40 con 0x7f04281043a0 2023-07-12T15:27:31.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.827+0000 7f041effd700 1 -- 172.21.15.69:0/1376984558 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+11454 (secure 0 0 0) 0x7f0420059280 con 0x7f04281043a0 2023-07-12T15:27:31.829 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:27:31.829 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":58,"fsid":"dc75a490-20c7-11ee-9b31-001a4aab830c","created":"2023-07-12T15:22:48.769414+0000","modified":"2023-07-12T15:27:30.556918+0000","last_up_change":"2023-07-12T15:27:26.534810+0000","last_in_change":"2023-07-12T15:27:10.083296+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-12T15:25:39.745272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ca84d141-6fa9-46e6-9fa4-2b6ebe1d0cad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6801","nonce":657400327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":657400327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":657400327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":657400327}]},"public_addr":"172.21.15.69:6801/657400327","cluster_addr":"172.21.15.69:6802/657400327","heartbeat_back_addr":"172.21.15.69:6804/657400327","heartbeat_front_addr":"172.21.15.69:6803/657400327","state":["exists","up"]},{"osd":1,"uuid":"6e1e1865-07a1-4490-af3b-48f832fe203e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6805","nonce":2434701037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2434701037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2434701037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2434701037}]},"public_addr":"172.21.15.69:6805/2434701037","cluster_addr":"172.21.15.69:6806/2434701037","heartbeat_back_addr":"172.21.15.69:6808/2434701037","heartbeat_front_addr":"172.21.15.69:6807/2434701037","state":["exists","up"]},{"osd":2,"uuid":"67ce3cb4-9ddd-477c-a490-dd919b0cf1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6809","nonce":1319832966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1319832966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1319832966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1319832966}]},"public_addr":"172.21.15.69:6809/1319832966","cluster_addr":"172.21.15.69:6810/1319832966","heartbeat_back_addr":"172.21.15.69:6812/1319832966","heartbeat_front_addr":"172.21.15.69:6811/1319832966","state":["exists","up"]},{"osd":3,"uuid":"0a5e5d11-ad42-4663-8c6b-d04e9b968c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6813","nonce":2610582937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2610582937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2610582937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":2610582937}]},"public_addr":"172.21.15.69:6813/2610582937","cluster_addr":"172.21.15.69:6814/2610582937","heartbeat_back_addr":"172.21.15.69:6816/2610582937","heartbeat_front_addr":"172.21.15.69:6815/2610582937","state":["exists","up"]},{"osd":4,"uuid":"672b53e3-659e-408b-98e5-a3cbaf549b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":184109944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6801","nonce":184109944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":184109944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":184109944}]},"public_addr":"172.21.15.145:6800/184109944","cluster_addr":"172.21.15.145:6801/184109944","heartbeat_back_addr":"172.21.15.145:6803/184109944","heartbeat_front_addr":"172.21.15.145:6802/184109944","state":["exists","up"]},{"osd":5,"uuid":"975b3691-5232-4774-9b74-087ae97745a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":363217135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6805","nonce":363217135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":363217135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":363217135}]},"public_addr":"172.21.15.145:6804/363217135","cluster_addr":"172.21.15.145:6805/363217135","heartbeat_back_addr":"172.21.15.145:6807/363217135","heartbeat_front_addr":"172.21.15.145:6806/363217135","state":["exists","up"]},{"osd":6,"uuid":"2aa56dbc-dd4c-487c-8f74-00b05022b978","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":3456495522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6809","nonce":3456495522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":3456495522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":3456495522}]},"public_addr":"172.21.15.145:6808/3456495522","cluster_addr":"172.21.15.145:6809/3456495522","heartbeat_back_addr":"172.21.15.145:6811/3456495522","heartbeat_front_addr":"172.21.15.145:6810/3456495522","state":["exists","up"]},{"osd":7,"uuid":"55f492ca-7e77-4643-a798-b0899a6bf944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2623299546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6813","nonce":2623299546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6815","nonce":2623299546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2623299546}]},"public_addr":"172.21.15.145:6812/2623299546","cluster_addr":"172.21.15.145:6813/2623299546","heartbeat_back_addr":"172.21.15.145:6815/2623299546","heartbeat_front_addr":"172.21.15.145:6814/2623299546","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:24:55.893488+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:25:16.619394+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:25:37.932802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:25:59.145227+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:26:19.105589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:26:41.005568+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:27:03.198657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:27:24.827885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/3740256751":"2023-07-13T15:23:43.423483+0000","172.21.15.69:6800/339337620":"2023-07-13T15:23:43.423483+0000","172.21.15.69:0/3398378951":"2023-07-13T15:23:21.142752+0000","172.21.15.69:0/2429471300":"2023-07-13T15:23:43.423483+0000","172.21.15.69:0/2792247380":"2023-07-13T15:23:21.142752+0000","172.21.15.69:6800/654231506":"2023-07-13T15:23:21.142752+0000","172.21.15.69:0/1688014036":"2023-07-13T15:23:21.142752+0000","172.21.15.69:0/2361078360":"2023-07-13T15:23:43.423483+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-12T15:27:31.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.829+0000 7f04301a6700 1 -- 172.21.15.69:0/1376984558 >> v2:172.21.15.69:6800/284433772 conn(0x7f041406c720 msgr2=0x7f041406ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:31.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.829+0000 7f04301a6700 1 --2- 172.21.15.69:0/1376984558 >> v2:172.21.15.69:6800/284433772 conn(0x7f041406c720 0x7f041406ebd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0418009de0 tx=0x7f0418009190 comp rx=0 tx=0).stop 2023-07-12T15:27:31.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.829+0000 7f04301a6700 1 -- 172.21.15.69:0/1376984558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04281043a0 msgr2=0x7f042819a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:31.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.829+0000 7f04301a6700 1 --2- 172.21.15.69:0/1376984558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04281043a0 0x7f042819a700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f042000c340 tx=0x7f042000c650 comp rx=0 tx=0).stop 2023-07-12T15:27:31.831 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.829+0000 7f04301a6700 1 -- 172.21.15.69:0/1376984558 shutdown_connections 2023-07-12T15:27:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.829+0000 7f04301a6700 1 --2- 172.21.15.69:0/1376984558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04281043a0 0x7f042819a700 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.829+0000 7f04301a6700 1 --2- 172.21.15.69:0/1376984558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04280ff300 0x7f0428195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.829+0000 7f04301a6700 1 --2- 172.21.15.69:0/1376984558 >> v2:172.21.15.69:6800/284433772 conn(0x7f041406c720 0x7f041406ebd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.829+0000 7f04301a6700 1 --2- 172.21.15.69:0/1376984558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04280ffca0 0x7f0428196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.829+0000 7f04301a6700 1 -- 172.21.15.69:0/1376984558 >> 172.21.15.69:0/1376984558 conn(0x7f0428076160 msgr2=0x7f04280fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.830+0000 7f04301a6700 1 -- 172.21.15.69:0/1376984558 shutdown_connections 2023-07-12T15:27:31.832 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:31.830+0000 7f04301a6700 1 -- 172.21.15.69:0/1376984558 wait complete. 2023-07-12T15:27:31.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:31 smithi145 ceph-mon[95801]: osdmap e58: 8 total, 8 up, 8 in 2023-07-12T15:27:32.222 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-12T15:25:39.745272+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-12T15:27:32.223 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-12T15:27:32.776 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:27:32.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:32 smithi145 ceph-mon[95801]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-12T15:27:32.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1376984558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-12T15:27:32.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:32 smithi069 ceph-mon[102777]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-12T15:27:32.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1376984558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-12T15:27:32.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:32 smithi069 ceph-mon[94241]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-12T15:27:32.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1376984558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-12T15:27:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.101+0000 7f59a9968700 1 -- 172.21.15.69:0/1828313771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59a41012c0 msgr2=0x7f59a4101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.101+0000 7f59a9968700 1 --2- 172.21.15.69:0/1828313771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59a41012c0 0x7f59a4101690 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5998009990 tx=0x7f5998009ca0 comp rx=0 tx=0).stop 2023-07-12T15:27:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.101+0000 7f59a9968700 1 -- 172.21.15.69:0/1828313771 shutdown_connections 2023-07-12T15:27:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.101+0000 7f59a9968700 1 --2- 172.21.15.69:0/1828313771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59a410a7e0 0x7f59a410cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:34.103 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.101+0000 7f59a9968700 1 --2- 172.21.15.69:0/1828313771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59a41012c0 0x7f59a4101690 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:34.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.101+0000 7f59a9968700 1 --2- 172.21.15.69:0/1828313771 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a4101bd0 0x7f59a410a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:34.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.101+0000 7f59a9968700 1 -- 172.21.15.69:0/1828313771 >> 172.21.15.69:0/1828313771 conn(0x7f59a4075f70 msgr2=0x7f59a4076370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:34.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.101+0000 7f59a9968700 1 -- 172.21.15.69:0/1828313771 shutdown_connections 2023-07-12T15:27:34.104 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.101+0000 7f59a9968700 1 -- 172.21.15.69:0/1828313771 wait complete. 2023-07-12T15:27:34.105 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.102+0000 7f59a9968700 1 Processor -- start 2023-07-12T15:27:34.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.102+0000 7f59a9968700 1 -- start start 2023-07-12T15:27:34.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.102+0000 7f59a9968700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a41012c0 0x7f59a419b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:34.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a9968700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59a4101bd0 0x7f59a419bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:34.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a9968700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59a410a7e0 0x7f59a4195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:34.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a9968700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a4195e70 con 0x7f59a410a7e0 2023-07-12T15:27:34.106 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a9968700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a4195fe0 con 0x7f59a41012c0 2023-07-12T15:27:34.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a9968700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a41961a0 con 0x7f59a4101bd0 2023-07-12T15:27:34.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a27fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59a4101bd0 0x7f59a419bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:34.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a37fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59a410a7e0 0x7f59a4195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:34.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a27fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59a4101bd0 0x7f59a419bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:33834/0 (socket says 172.21.15.69:33834) 2023-07-12T15:27:34.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a27fc700 1 -- 172.21.15.69:0/926846837 learned_addr learned my addr 172.21.15.69:0/926846837 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:27:34.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a2ffd700 1 --2- 172.21.15.69:0/926846837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a41012c0 0x7f59a419b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:34.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a37fe700 1 -- 172.21.15.69:0/926846837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59a4101bd0 msgr2=0x7f59a419bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:34.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a37fe700 1 --2- 172.21.15.69:0/926846837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59a4101bd0 0x7f59a419bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:34.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a37fe700 1 -- 172.21.15.69:0/926846837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a41012c0 msgr2=0x7f59a419b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:34.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a37fe700 1 --2- 172.21.15.69:0/926846837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a41012c0 0x7f59a419b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:34.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a37fe700 1 -- 172.21.15.69:0/926846837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59980095f0 con 0x7f59a410a7e0 2023-07-12T15:27:34.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a27fc700 1 --2- 172.21.15.69:0/926846837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59a4101bd0 0x7f59a419bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:27:34.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a37fe700 1 --2- 172.21.15.69:0/926846837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59a410a7e0 0x7f59a4195800 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f599c00ef90 tx=0x7f599c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:34.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a8966700 1 -- 172.21.15.69:0/926846837 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c018070 con 0x7f59a410a7e0 2023-07-12T15:27:34.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a8966700 1 -- 172.21.15.69:0/926846837 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f599c00f040 con 0x7f59a410a7e0 2023-07-12T15:27:34.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a9968700 1 -- 172.21.15.69:0/926846837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a4196420 con 0x7f59a410a7e0 2023-07-12T15:27:34.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a8966700 1 -- 172.21.15.69:0/926846837 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c00cc50 con 0x7f59a410a7e0 2023-07-12T15:27:34.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.103+0000 7f59a9968700 1 -- 172.21.15.69:0/926846837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a41a2770 con 0x7f59a410a7e0 2023-07-12T15:27:34.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.104+0000 7f59a9968700 1 -- 172.21.15.69:0/926846837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a40686f0 con 0x7f59a410a7e0 2023-07-12T15:27:34.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.105+0000 7f59a8966700 1 -- 172.21.15.69:0/926846837 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 15) v1 ==== 89784+0+0 (secure 0 0 0) 0x7f599c003a00 con 0x7f59a410a7e0 2023-07-12T15:27:34.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.105+0000 7f59a8966700 1 --2- 172.21.15.69:0/926846837 >> v2:172.21.15.69:6800/284433772 conn(0x7f598c06c720 0x7f598c06ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:34.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.105+0000 7f59a8966700 1 -- 172.21.15.69:0/926846837 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (secure 0 0 0) 0x7f599c08af30 con 0x7f59a410a7e0 2023-07-12T15:27:34.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.105+0000 7f59a2ffd700 1 --2- 172.21.15.69:0/926846837 >> v2:172.21.15.69:6800/284433772 conn(0x7f598c06c720 0x7f598c06ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:34.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.106+0000 7f59a2ffd700 1 --2- 172.21.15.69:0/926846837 >> v2:172.21.15.69:6800/284433772 conn(0x7f598c06c720 0x7f598c06ebd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f599800b2e0 tx=0x7f5998005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:34.111 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.110+0000 7f59a8966700 1 -- 172.21.15.69:0/926846837 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f599c055b70 con 0x7f59a410a7e0 2023-07-12T15:27:34.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.297+0000 7f59a9968700 1 -- 172.21.15.69:0/926846837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f59a41a2900 con 0x7f59a410a7e0 2023-07-12T15:27:34.298 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.297+0000 7f59a8966700 1 -- 172.21.15.69:0/926846837 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f599c059190 con 0x7f59a410a7e0 2023-07-12T15:27:34.299 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2023-07-12T15:27:34.300 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.299+0000 7f59a9968700 1 -- 172.21.15.69:0/926846837 >> v2:172.21.15.69:6800/284433772 conn(0x7f598c06c720 msgr2=0x7f598c06ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:34.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.299+0000 7f59a9968700 1 --2- 172.21.15.69:0/926846837 >> v2:172.21.15.69:6800/284433772 conn(0x7f598c06c720 0x7f598c06ebd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f599800b2e0 tx=0x7f5998005fd0 comp rx=0 tx=0).stop 2023-07-12T15:27:34.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.299+0000 7f59a9968700 1 -- 172.21.15.69:0/926846837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59a410a7e0 msgr2=0x7f59a4195800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:34.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.299+0000 7f59a9968700 1 --2- 172.21.15.69:0/926846837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59a410a7e0 0x7f59a4195800 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f599c00ef90 tx=0x7f599c00c3a0 comp rx=0 tx=0).stop 2023-07-12T15:27:34.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.299+0000 7f59a9968700 1 -- 172.21.15.69:0/926846837 shutdown_connections 2023-07-12T15:27:34.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.299+0000 7f59a9968700 1 --2- 172.21.15.69:0/926846837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59a410a7e0 0x7f59a4195800 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:34.301 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.299+0000 7f59a9968700 1 --2- 172.21.15.69:0/926846837 >> v2:172.21.15.69:6800/284433772 conn(0x7f598c06c720 0x7f598c06ebd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:34.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.299+0000 7f59a9968700 1 --2- 172.21.15.69:0/926846837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a41012c0 0x7f59a419b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:34.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.299+0000 7f59a9968700 1 --2- 172.21.15.69:0/926846837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59a4101bd0 0x7f59a419bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:34.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.299+0000 7f59a9968700 1 -- 172.21.15.69:0/926846837 >> 172.21.15.69:0/926846837 conn(0x7f59a4075f70 msgr2=0x7f59a40fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:34.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.299+0000 7f59a9968700 1 -- 172.21.15.69:0/926846837 shutdown_connections 2023-07-12T15:27:34.302 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:27:34.299+0000 7f59a9968700 1 -- 172.21.15.69:0/926846837 wait complete. 2023-07-12T15:27:34.695 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:34 smithi069 ceph-mon[102777]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2023-07-12T15:27:34.696 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/926846837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-12T15:27:34.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:34 smithi069 ceph-mon[94241]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2023-07-12T15:27:34.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/926846837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-12T15:27:34.730 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi069 2023-07-12T15:27:34.731 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch apply rgw foo.a --placement '1;smithi069=foo.a' 2023-07-12T15:27:34.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:34 smithi145 ceph-mon[95801]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2023-07-12T15:27:34.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/926846837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-12T15:27:35.300 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:27:36.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:35 smithi069 ceph-mon[94241]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-07-12T15:27:36.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:35 smithi069 ceph-mon[102777]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-07-12T15:27:36.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:35 smithi145 ceph-mon[95801]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-07-12T15:27:36.640 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.636+0000 7f002df4f700 1 -- 172.21.15.145:0/1619967075 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00280ff070 msgr2=0x7f00280ff480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:36.640 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.636+0000 7f002df4f700 1 --2- 172.21.15.145:0/1619967075 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00280ff070 0x7f00280ff480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0018009990 tx=0x7f0018009ca0 comp rx=0 tx=0).stop 2023-07-12T15:27:36.640 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.636+0000 7f002df4f700 1 -- 172.21.15.145:0/1619967075 shutdown_connections 2023-07-12T15:27:36.641 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.636+0000 7f002df4f700 1 --2- 172.21.15.145:0/1619967075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0028100c00 0x7f00281074c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:36.641 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.636+0000 7f002df4f700 1 --2- 172.21.15.145:0/1619967075 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00280ff070 0x7f00280ff480 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:36.641 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.636+0000 7f002df4f700 1 --2- 172.21.15.145:0/1619967075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0028100270 0x7f00281006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:36.641 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.636+0000 7f002df4f700 1 -- 172.21.15.145:0/1619967075 >> 172.21.15.145:0/1619967075 conn(0x7f00280fa6a0 msgr2=0x7f00280fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:36.641 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.636+0000 7f002df4f700 1 -- 172.21.15.145:0/1619967075 shutdown_connections 2023-07-12T15:27:36.642 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.636+0000 7f002df4f700 1 -- 172.21.15.145:0/1619967075 wait complete. 2023-07-12T15:27:36.642 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.636+0000 7f002df4f700 1 Processor -- start 2023-07-12T15:27:36.642 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.636+0000 7f002df4f700 1 -- start start 2023-07-12T15:27:36.642 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f002df4f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00280ff070 0x7f0028103000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:36.642 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f002df4f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028100270 0x7f0028101650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:36.642 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f002df4f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0028100c00 0x7f0028101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:36.642 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f002df4f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0028102310 con 0x7f0028100c00 2023-07-12T15:27:36.643 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f002df4f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0028102480 con 0x7f0028100270 2023-07-12T15:27:36.643 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f002df4f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002819f620 con 0x7f00280ff070 2023-07-12T15:27:36.643 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f0026ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028100270 0x7f0028101650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:36.643 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f0026ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028100270 0x7f0028101650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44212/0 (socket says 172.21.15.145:44212) 2023-07-12T15:27:36.643 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f0026ffd700 1 -- 172.21.15.145:0/1691532120 learned_addr learned my addr 172.21.15.145:0/1691532120 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:27:36.643 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f0027fff700 1 --2- 172.21.15.145:0/1691532120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0028100c00 0x7f0028101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:36.643 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f00277fe700 1 --2- 172.21.15.145:0/1691532120 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00280ff070 0x7f0028103000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:36.644 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f0026ffd700 1 -- 172.21.15.145:0/1691532120 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00280ff070 msgr2=0x7f0028103000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:36.644 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f0026ffd700 1 --2- 172.21.15.145:0/1691532120 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00280ff070 0x7f0028103000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:36.644 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f0026ffd700 1 -- 172.21.15.145:0/1691532120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0028100c00 msgr2=0x7f0028101b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:36.644 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f0026ffd700 1 --2- 172.21.15.145:0/1691532120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0028100c00 0x7f0028101b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:36.644 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f0026ffd700 1 -- 172.21.15.145:0/1691532120 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00180095f0 con 0x7f0028100270 2023-07-12T15:27:36.645 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f0026ffd700 1 --2- 172.21.15.145:0/1691532120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028100270 0x7f0028101650 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f001c00d980 tx=0x7f001c00dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:36.645 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f0024ff9700 1 -- 172.21.15.145:0/1691532120 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c00bb60 con 0x7f0028100270 2023-07-12T15:27:36.645 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f002df4f700 1 -- 172.21.15.145:0/1691532120 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002819f900 con 0x7f0028100270 2023-07-12T15:27:36.645 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f0024ff9700 1 -- 172.21.15.145:0/1691532120 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f001c007640 con 0x7f0028100270 2023-07-12T15:27:36.645 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f0024ff9700 1 -- 172.21.15.145:0/1691532120 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c004d40 con 0x7f0028100270 2023-07-12T15:27:36.645 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.637+0000 7f002df4f700 1 -- 172.21.15.145:0/1691532120 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002819fe50 con 0x7f0028100270 2023-07-12T15:27:36.645 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.638+0000 7f002df4f700 1 -- 172.21.15.145:0/1691532120 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0028109f80 con 0x7f0028100270 2023-07-12T15:27:36.647 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.639+0000 7f0024ff9700 1 -- 172.21.15.145:0/1691532120 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 15) v1 ==== 89784+0+0 (secure 0 0 0) 0x7f001c004020 con 0x7f0028100270 2023-07-12T15:27:36.647 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.639+0000 7f0024ff9700 1 --2- 172.21.15.145:0/1691532120 >> v2:172.21.15.69:6800/284433772 conn(0x7f001006c720 0x7f001006ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:36.647 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.639+0000 7f0024ff9700 1 -- 172.21.15.145:0/1691532120 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (secure 0 0 0) 0x7f001c08b1a0 con 0x7f0028100270 2023-07-12T15:27:36.647 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.639+0000 7f00277fe700 1 --2- 172.21.15.145:0/1691532120 >> v2:172.21.15.69:6800/284433772 conn(0x7f001006c720 0x7f001006ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:36.647 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.640+0000 7f00277fe700 1 --2- 172.21.15.145:0/1691532120 >> v2:172.21.15.69:6800/284433772 conn(0x7f001006c720 0x7f001006ebd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0018006010 tx=0x7f001800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:36.648 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.644+0000 7f0024ff9700 1 -- 172.21.15.145:0/1691532120 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f001c055e90 con 0x7f0028100270 2023-07-12T15:27:36.839 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.837+0000 7f002df4f700 1 -- 172.21.15.145:0/1691532120 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi069=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f0028068d70 con 0x7f001006c720 2023-07-12T15:27:36.846 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.845+0000 7f0024ff9700 1 -- 172.21.15.145:0/1691532120 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (secure 0 0 0) 0x7f0028068d70 con 0x7f001006c720 2023-07-12T15:27:36.847 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled rgw.foo.a update... 2023-07-12T15:27:36.849 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.847+0000 7f002df4f700 1 -- 172.21.15.145:0/1691532120 >> v2:172.21.15.69:6800/284433772 conn(0x7f001006c720 msgr2=0x7f001006ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:36.849 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.847+0000 7f002df4f700 1 --2- 172.21.15.145:0/1691532120 >> v2:172.21.15.69:6800/284433772 conn(0x7f001006c720 0x7f001006ebd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0018006010 tx=0x7f001800b540 comp rx=0 tx=0).stop 2023-07-12T15:27:36.849 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.847+0000 7f002df4f700 1 -- 172.21.15.145:0/1691532120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028100270 msgr2=0x7f0028101650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:36.849 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.847+0000 7f002df4f700 1 --2- 172.21.15.145:0/1691532120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028100270 0x7f0028101650 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f001c00d980 tx=0x7f001c00dd40 comp rx=0 tx=0).stop 2023-07-12T15:27:36.849 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.847+0000 7f002df4f700 1 -- 172.21.15.145:0/1691532120 shutdown_connections 2023-07-12T15:27:36.849 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.847+0000 7f002df4f700 1 --2- 172.21.15.145:0/1691532120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0028100c00 0x7f0028101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:36.850 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.847+0000 7f002df4f700 1 --2- 172.21.15.145:0/1691532120 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00280ff070 0x7f0028103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:36.850 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.847+0000 7f002df4f700 1 --2- 172.21.15.145:0/1691532120 >> v2:172.21.15.69:6800/284433772 conn(0x7f001006c720 0x7f001006ebd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:36.852 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.847+0000 7f002df4f700 1 --2- 172.21.15.145:0/1691532120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028100270 0x7f0028101650 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:36.852 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.847+0000 7f002df4f700 1 -- 172.21.15.145:0/1691532120 >> 172.21.15.145:0/1691532120 conn(0x7f00280fa6a0 msgr2=0x7f0028068660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:36.852 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.847+0000 7f002df4f700 1 -- 172.21.15.145:0/1691532120 shutdown_connections 2023-07-12T15:27:36.852 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:36.847+0000 7f002df4f700 1 -- 172.21.15.145:0/1691532120 wait complete. 2023-07-12T15:27:37.244 DEBUG:teuthology.orchestra.run.smithi069:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@rgw.foo.a.service 2023-07-12T15:27:37.247 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi145 2023-07-12T15:27:37.247 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd pool create datapool 3 3 replicated 2023-07-12T15:27:37.277 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:-- Logs begin at Wed 2023-07-12 15:05:12 UTC. -- 2023-07-12T15:27:37.795 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:27:38.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:37 smithi145 ceph-mon[95801]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi069=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:37 smithi145 ceph-mon[95801]: Saving service rgw.foo.a spec with placement smithi069=foo.a;count:1 2023-07-12T15:27:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:38.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-12T15:27:38.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-12T15:27:38.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:38.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:37 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:38.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:37 smithi145 ceph-mon[95801]: Deploying daemon rgw.foo.a on smithi069 2023-07-12T15:27:38.161 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[94241]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi069=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:38.161 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[94241]: Saving service rgw.foo.a spec with placement smithi069=foo.a;count:1 2023-07-12T15:27:38.161 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:38.161 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:38.161 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:38.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:38.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:38.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-12T15:27:38.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-12T15:27:38.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:38.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:38.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[94241]: Deploying daemon rgw.foo.a on smithi069 2023-07-12T15:27:38.163 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[102777]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi069=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:38.163 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[102777]: Saving service rgw.foo.a spec with placement smithi069=foo.a;count:1 2023-07-12T15:27:38.163 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:38.163 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:38.163 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:38.164 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:38.164 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:38.164 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-07-12T15:27:38.164 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-07-12T15:27:38.164 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:38.164 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:38.164 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:37 smithi069 ceph-mon[102777]: Deploying daemon rgw.foo.a on smithi069 2023-07-12T15:27:38.485 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Jul 12 15:27:38 smithi069 systemd[1]: Starting Ceph rgw.foo.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:27:38.876 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Jul 12 15:27:38 smithi069 podman[124484]: 2023-07-12T15:27:38.877 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Jul 12 15:27:38 smithi069 bash[124484]: de5ca2967e8ea53f1e70c1177f89e0c37ec99e7b6a077aa3c6b6bffc1c79ca5d 2023-07-12T15:27:39.032 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.029+0000 7fe7e5485700 1 -- 172.21.15.145:0/2460978752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e00ff070 msgr2=0x7fe7e00ff480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:39.032 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.029+0000 7fe7e5485700 1 --2- 172.21.15.145:0/2460978752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e00ff070 0x7fe7e00ff480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009990 tx=0x7fe7d4009ca0 comp rx=0 tx=0).stop 2023-07-12T15:27:39.032 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.029+0000 7fe7e5485700 1 -- 172.21.15.145:0/2460978752 shutdown_connections 2023-07-12T15:27:39.033 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.029+0000 7fe7e5485700 1 --2- 172.21.15.145:0/2460978752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0100c00 0x7fe7e01074c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:39.033 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.029+0000 7fe7e5485700 1 --2- 172.21.15.145:0/2460978752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e00ff070 0x7fe7e00ff480 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:39.033 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.029+0000 7fe7e5485700 1 --2- 172.21.15.145:0/2460978752 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e0100270 0x7fe7e01006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:39.033 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.029+0000 7fe7e5485700 1 -- 172.21.15.145:0/2460978752 >> 172.21.15.145:0/2460978752 conn(0x7fe7e00fa6a0 msgr2=0x7fe7e00fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:39.033 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.029+0000 7fe7e5485700 1 -- 172.21.15.145:0/2460978752 shutdown_connections 2023-07-12T15:27:39.033 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.029+0000 7fe7e5485700 1 -- 172.21.15.145:0/2460978752 wait complete. 2023-07-12T15:27:39.034 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.029+0000 7fe7e5485700 1 Processor -- start 2023-07-12T15:27:39.034 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7e5485700 1 -- start start 2023-07-12T15:27:39.034 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7e5485700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e00ff070 0x7fe7e0069180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:39.034 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7e5485700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e0100270 0x7fe7e019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:39.034 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7e5485700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0100c00 0x7fe7e019c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:39.034 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7e5485700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e019d0b0 con 0x7fe7e0100c00 2023-07-12T15:27:39.034 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7e5485700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e019d220 con 0x7fe7e0100270 2023-07-12T15:27:39.034 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7e5485700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e019d3e0 con 0x7fe7e00ff070 2023-07-12T15:27:39.035 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7de7fc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e0100270 0x7fe7e019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:39.035 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7de7fc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e0100270 0x7fe7e019a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44226/0 (socket says 172.21.15.145:44226) 2023-07-12T15:27:39.035 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7de7fc700 1 -- 172.21.15.145:0/3764167131 learned_addr learned my addr 172.21.15.145:0/3764167131 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:27:39.035 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7deffd700 1 --2- 172.21.15.145:0/3764167131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e00ff070 0x7fe7e0069180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:39.035 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7df7fe700 1 --2- 172.21.15.145:0/3764167131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0100c00 0x7fe7e019c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:39.035 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7deffd700 1 -- 172.21.15.145:0/3764167131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e0100270 msgr2=0x7fe7e019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:39.035 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7deffd700 1 --2- 172.21.15.145:0/3764167131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e0100270 0x7fe7e019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:39.036 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7deffd700 1 -- 172.21.15.145:0/3764167131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0100c00 msgr2=0x7fe7e019c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:39.036 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7deffd700 1 --2- 172.21.15.145:0/3764167131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0100c00 0x7fe7e019c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:39.036 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7deffd700 1 -- 172.21.15.145:0/3764167131 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d40095f0 con 0x7fe7e00ff070 2023-07-12T15:27:39.036 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7de7fc700 1 --2- 172.21.15.145:0/3764167131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e0100270 0x7fe7e019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:27:39.036 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.030+0000 7fe7deffd700 1 --2- 172.21.15.145:0/3764167131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e00ff070 0x7fe7e0069180 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4000c00 tx=0x7fe7d400bb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:39.036 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.031+0000 7fe7c7fff700 1 -- 172.21.15.145:0/3764167131 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d401d070 con 0x7fe7e00ff070 2023-07-12T15:27:39.036 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.031+0000 7fe7c7fff700 1 -- 172.21.15.145:0/3764167131 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fe7d4003d20 con 0x7fe7e00ff070 2023-07-12T15:27:39.037 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.031+0000 7fe7c7fff700 1 -- 172.21.15.145:0/3764167131 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4022db0 con 0x7fe7e00ff070 2023-07-12T15:27:39.037 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.031+0000 7fe7e5485700 1 -- 172.21.15.145:0/3764167131 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e019d660 con 0x7fe7e00ff070 2023-07-12T15:27:39.037 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.031+0000 7fe7e5485700 1 -- 172.21.15.145:0/3764167131 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e019da70 con 0x7fe7e00ff070 2023-07-12T15:27:39.038 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.032+0000 7fe7e5485700 1 -- 172.21.15.145:0/3764167131 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e004e800 con 0x7fe7e00ff070 2023-07-12T15:27:39.038 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.032+0000 7fe7c7fff700 1 -- 172.21.15.145:0/3764167131 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 15) v1 ==== 89784+0+0 (secure 0 0 0) 0x7fe7d4022470 con 0x7fe7e00ff070 2023-07-12T15:27:39.038 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.032+0000 7fe7c7fff700 1 --2- 172.21.15.145:0/3764167131 >> v2:172.21.15.69:6800/284433772 conn(0x7fe7c806c720 0x7fe7c806ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:39.038 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.032+0000 7fe7c7fff700 1 -- 172.21.15.145:0/3764167131 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (secure 0 0 0) 0x7fe7d408c740 con 0x7fe7e00ff070 2023-07-12T15:27:39.038 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.033+0000 7fe7de7fc700 1 --2- 172.21.15.145:0/3764167131 >> v2:172.21.15.69:6800/284433772 conn(0x7fe7c806c720 0x7fe7c806ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:39.039 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.033+0000 7fe7de7fc700 1 --2- 172.21.15.145:0/3764167131 >> v2:172.21.15.69:6800/284433772 conn(0x7fe7c806c720 0x7fe7c806ebd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0011550 tx=0x7fe7d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:39.039 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.037+0000 7fe7c7fff700 1 -- 172.21.15.145:0/3764167131 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe7d4057400 con 0x7fe7e00ff070 2023-07-12T15:27:39.233 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:39.232+0000 7fe7e5485700 1 -- 172.21.15.145:0/3764167131 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7fe7e019ddc0 con 0x7fe7e00ff070 2023-07-12T15:27:39.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:39 smithi069 ceph-mon[102777]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:27:39.235 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Jul 12 15:27:38 smithi069 systemd[1]: Started Ceph rgw.foo.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:27:39.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:39 smithi069 ceph-mon[94241]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:27:39.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:39 smithi145 ceph-mon[95801]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:27:40.119 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:40.117+0000 7fe7c7fff700 1 -- 172.21.15.145:0/3764167131 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v60) v1 ==== 160+0+0 (secure 0 0 0) 0x7fe7d405aa20 con 0x7fe7e00ff070 2023-07-12T15:27:40.126 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:40.124+0000 7fe7e5485700 1 -- 172.21.15.145:0/3764167131 >> v2:172.21.15.69:6800/284433772 conn(0x7fe7c806c720 msgr2=0x7fe7c806ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:40.126 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:40.124+0000 7fe7e5485700 1 --2- 172.21.15.145:0/3764167131 >> v2:172.21.15.69:6800/284433772 conn(0x7fe7c806c720 0x7fe7c806ebd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0011550 tx=0x7fe7d0009260 comp rx=0 tx=0).stop 2023-07-12T15:27:40.127 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:40.124+0000 7fe7e5485700 1 -- 172.21.15.145:0/3764167131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e00ff070 msgr2=0x7fe7e0069180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:40.127 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:40.124+0000 7fe7e5485700 1 --2- 172.21.15.145:0/3764167131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e00ff070 0x7fe7e0069180 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4000c00 tx=0x7fe7d400bb30 comp rx=0 tx=0).stop 2023-07-12T15:27:40.127 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:40.124+0000 7fe7e5485700 1 -- 172.21.15.145:0/3764167131 shutdown_connections 2023-07-12T15:27:40.127 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:40.124+0000 7fe7e5485700 1 --2- 172.21.15.145:0/3764167131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0100c00 0x7fe7e019c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:40.127 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:40.124+0000 7fe7e5485700 1 --2- 172.21.15.145:0/3764167131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e00ff070 0x7fe7e0069180 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:40.127 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:40.124+0000 7fe7e5485700 1 --2- 172.21.15.145:0/3764167131 >> v2:172.21.15.69:6800/284433772 conn(0x7fe7c806c720 0x7fe7c806ebd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:40.127 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:40.124+0000 7fe7e5485700 1 --2- 172.21.15.145:0/3764167131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e0100270 0x7fe7e019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:40.127 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:40.125+0000 7fe7e5485700 1 -- 172.21.15.145:0/3764167131 >> 172.21.15.145:0/3764167131 conn(0x7fe7e00fa6a0 msgr2=0x7fe7e00fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:40.128 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:40.125+0000 7fe7e5485700 1 -- 172.21.15.145:0/3764167131 shutdown_connections 2023-07-12T15:27:40.128 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:40.125+0000 7fe7e5485700 1 -- 172.21.15.145:0/3764167131 wait complete. 2023-07-12T15:27:40.137 INFO:teuthology.orchestra.run.smithi145.stderr:pool 'datapool' created 2023-07-12T15:27:40.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:40 smithi145 ceph-mon[95801]: Saving service rgw.foo.a spec with placement smithi069=foo.a;count:1 2023-07-12T15:27:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:40 smithi145 ceph-mon[95801]: osdmap e59: 8 total, 8 up, 8 in 2023-07-12T15:27:40.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4181607302' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-12T15:27:40.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:40 smithi145 ceph-mon[95801]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-12T15:27:40.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:40 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:40.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/3764167131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2023-07-12T15:27:40.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:40 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2023-07-12T15:27:40.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:40 smithi145 ceph-mon[95801]: pgmap v167: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:27:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[102777]: Saving service rgw.foo.a spec with placement smithi069=foo.a;count:1 2023-07-12T15:27:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[102777]: osdmap e59: 8 total, 8 up, 8 in 2023-07-12T15:27:40.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4181607302' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-12T15:27:40.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[102777]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-12T15:27:40.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:40.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/3764167131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2023-07-12T15:27:40.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2023-07-12T15:27:40.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[102777]: pgmap v167: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:27:40.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[94241]: Saving service rgw.foo.a spec with placement smithi069=foo.a;count:1 2023-07-12T15:27:40.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:40.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[94241]: osdmap e59: 8 total, 8 up, 8 in 2023-07-12T15:27:40.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4181607302' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-12T15:27:40.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[94241]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-07-12T15:27:40.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:40.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/3764167131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2023-07-12T15:27:40.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2023-07-12T15:27:40.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:40 smithi069 ceph-mon[94241]: pgmap v167: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:27:40.561 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- rbd pool init datapool 2023-07-12T15:27:41.112 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:27:41.379 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:41 smithi145 ceph-mon[95801]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-07-12T15:27:41.380 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:41 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2023-07-12T15:27:41.380 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:41 smithi145 ceph-mon[95801]: osdmap e60: 8 total, 8 up, 8 in 2023-07-12T15:27:41.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:41 smithi069 ceph-mon[102777]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-07-12T15:27:41.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:41 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2023-07-12T15:27:41.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:41 smithi069 ceph-mon[102777]: osdmap e60: 8 total, 8 up, 8 in 2023-07-12T15:27:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:41 smithi069 ceph-mon[94241]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-07-12T15:27:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:41 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2023-07-12T15:27:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:41 smithi069 ceph-mon[94241]: osdmap e60: 8 total, 8 up, 8 in 2023-07-12T15:27:42.169 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:42 smithi145 ceph-mon[95801]: osdmap e61: 8 total, 8 up, 8 in 2023-07-12T15:27:42.170 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4181607302' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-07-12T15:27:42.170 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:42 smithi145 ceph-mon[95801]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-07-12T15:27:42.170 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:42 smithi145 ceph-mon[95801]: pgmap v170: 68 pgs: 6 creating+peering, 11 active+clean, 51 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2023-07-12T15:27:42.170 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:42 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:42.171 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:42 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:42.171 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:42 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:42.171 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:42 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:42.172 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:42 smithi145 ceph-mon[95801]: Checking dashboard <-> RGW credentials 2023-07-12T15:27:42.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[102777]: osdmap e61: 8 total, 8 up, 8 in 2023-07-12T15:27:42.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4181607302' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-07-12T15:27:42.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[102777]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-07-12T15:27:42.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[102777]: pgmap v170: 68 pgs: 6 creating+peering, 11 active+clean, 51 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2023-07-12T15:27:42.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:42.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:42.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:42.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:42.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[102777]: Checking dashboard <-> RGW credentials 2023-07-12T15:27:42.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[94241]: osdmap e61: 8 total, 8 up, 8 in 2023-07-12T15:27:42.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4181607302' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-07-12T15:27:42.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[94241]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-07-12T15:27:42.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[94241]: pgmap v170: 68 pgs: 6 creating+peering, 11 active+clean, 51 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2023-07-12T15:27:42.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:42.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:42.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:42.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:42.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:42 smithi069 ceph-mon[94241]: Checking dashboard <-> RGW credentials 2023-07-12T15:27:43.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:43 smithi069 ceph-mon[94241]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2023-07-12T15:27:43.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:43 smithi069 ceph-mon[94241]: osdmap e62: 8 total, 8 up, 8 in 2023-07-12T15:27:43.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/1914649101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2023-07-12T15:27:43.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:43 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2023-07-12T15:27:43.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:43 smithi069 ceph-mon[102777]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2023-07-12T15:27:43.534 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:43 smithi069 ceph-mon[102777]: osdmap e62: 8 total, 8 up, 8 in 2023-07-12T15:27:43.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/1914649101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2023-07-12T15:27:43.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:43 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2023-07-12T15:27:43.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:43 smithi145 ceph-mon[95801]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2023-07-12T15:27:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:43 smithi145 ceph-mon[95801]: osdmap e62: 8 total, 8 up, 8 in 2023-07-12T15:27:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/1914649101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2023-07-12T15:27:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:43 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2023-07-12T15:27:44.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2023-07-12T15:27:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[94241]: osdmap e63: 8 total, 8 up, 8 in 2023-07-12T15:27:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4181607302' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-07-12T15:27:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3042173169' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-07-12T15:27:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[94241]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-07-12T15:27:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[94241]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-07-12T15:27:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[94241]: pgmap v173: 100 pgs: 13 creating+peering, 28 active+clean, 59 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2023-07-12T15:27:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:44.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[94241]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2023-07-12T15:27:44.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[94241]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2023-07-12T15:27:44.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[94241]: osdmap e64: 8 total, 8 up, 8 in 2023-07-12T15:27:44.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2023-07-12T15:27:44.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[102777]: osdmap e63: 8 total, 8 up, 8 in 2023-07-12T15:27:44.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4181607302' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-07-12T15:27:44.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3042173169' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-07-12T15:27:44.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[102777]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-07-12T15:27:44.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[102777]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-07-12T15:27:44.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[102777]: pgmap v173: 100 pgs: 13 creating+peering, 28 active+clean, 59 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2023-07-12T15:27:44.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:44.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[102777]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2023-07-12T15:27:44.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[102777]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2023-07-12T15:27:44.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:44 smithi069 ceph-mon[102777]: osdmap e64: 8 total, 8 up, 8 in 2023-07-12T15:27:44.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:44 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2023-07-12T15:27:44.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:44 smithi145 ceph-mon[95801]: osdmap e63: 8 total, 8 up, 8 in 2023-07-12T15:27:44.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4181607302' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-07-12T15:27:44.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3042173169' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-07-12T15:27:44.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:44 smithi145 ceph-mon[95801]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-07-12T15:27:44.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:44 smithi145 ceph-mon[95801]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-07-12T15:27:44.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:44 smithi145 ceph-mon[95801]: pgmap v173: 100 pgs: 13 creating+peering, 28 active+clean, 59 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2023-07-12T15:27:44.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:44 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:44.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:44 smithi145 ceph-mon[95801]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2023-07-12T15:27:44.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:44 smithi145 ceph-mon[95801]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2023-07-12T15:27:44.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:44 smithi145 ceph-mon[95801]: osdmap e64: 8 total, 8 up, 8 in 2023-07-12T15:27:45.572 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.145 --placement '1;smithi145=iscsi.a' 2023-07-12T15:27:46.138 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:27:46.400 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:46 smithi145 ceph-mon[95801]: osdmap e65: 8 total, 8 up, 8 in 2023-07-12T15:27:46.400 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2106617291' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-07-12T15:27:46.400 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-07-12T15:27:46.400 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2888015557' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-07-12T15:27:46.400 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:46 smithi145 ceph-mon[95801]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-07-12T15:27:46.401 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:46 smithi145 ceph-mon[95801]: pgmap v176: 132 pgs: 10 creating+peering, 52 active+clean, 70 unknown; 578 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:27:46.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:46 smithi069 ceph-mon[94241]: osdmap e65: 8 total, 8 up, 8 in 2023-07-12T15:27:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2106617291' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-07-12T15:27:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-07-12T15:27:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2888015557' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-07-12T15:27:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:46 smithi069 ceph-mon[94241]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-07-12T15:27:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:46 smithi069 ceph-mon[94241]: pgmap v176: 132 pgs: 10 creating+peering, 52 active+clean, 70 unknown; 578 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:27:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:46 smithi069 ceph-mon[102777]: osdmap e65: 8 total, 8 up, 8 in 2023-07-12T15:27:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2106617291' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-07-12T15:27:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-07-12T15:27:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2888015557' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-07-12T15:27:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:46 smithi069 ceph-mon[102777]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-07-12T15:27:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:46 smithi069 ceph-mon[102777]: pgmap v176: 132 pgs: 10 creating+peering, 52 active+clean, 70 unknown; 578 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:27:47.270 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2023-07-12T15:27:47.270 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:47 smithi145 ceph-mon[95801]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2023-07-12T15:27:47.270 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:47 smithi145 ceph-mon[95801]: osdmap e66: 8 total, 8 up, 8 in 2023-07-12T15:27:47.271 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2888015557' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-12T15:27:47.271 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2106617291' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-12T15:27:47.271 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:47 smithi145 ceph-mon[95801]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-12T15:27:47.271 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-12T15:27:47.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2023-07-12T15:27:47.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[94241]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2023-07-12T15:27:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[94241]: osdmap e66: 8 total, 8 up, 8 in 2023-07-12T15:27:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2888015557' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-12T15:27:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2106617291' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-12T15:27:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[94241]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-12T15:27:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-12T15:27:47.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2023-07-12T15:27:47.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[102777]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2023-07-12T15:27:47.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[102777]: osdmap e66: 8 total, 8 up, 8 in 2023-07-12T15:27:47.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2888015557' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-12T15:27:47.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2106617291' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-12T15:27:47.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[102777]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-12T15:27:47.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-07-12T15:27:47.490 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.488+0000 7f0d95b30700 1 -- 172.21.15.145:0/700302930 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d901003d0 msgr2=0x7f0d90106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:47.490 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.488+0000 7f0d95b30700 1 --2- 172.21.15.145:0/700302930 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d901003d0 0x7f0d90106c90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800b0a0 tx=0x7f0d8800b3b0 comp rx=0 tx=0).stop 2023-07-12T15:27:47.490 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.488+0000 7f0d95b30700 1 -- 172.21.15.145:0/700302930 shutdown_connections 2023-07-12T15:27:47.490 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.488+0000 7f0d95b30700 1 --2- 172.21.15.145:0/700302930 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d901003d0 0x7f0d90106c90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:47.490 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.488+0000 7f0d95b30700 1 --2- 172.21.15.145:0/700302930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d900fe840 0x7f0d900fec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:47.490 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.488+0000 7f0d95b30700 1 --2- 172.21.15.145:0/700302930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d900ffa40 0x7f0d900ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:47.491 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.488+0000 7f0d95b30700 1 -- 172.21.15.145:0/700302930 >> 172.21.15.145:0/700302930 conn(0x7f0d900f9e70 msgr2=0x7f0d900fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:47.491 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.488+0000 7f0d95b30700 1 -- 172.21.15.145:0/700302930 shutdown_connections 2023-07-12T15:27:47.491 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.488+0000 7f0d95b30700 1 -- 172.21.15.145:0/700302930 wait complete. 2023-07-12T15:27:47.493 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.489+0000 7f0d95b30700 1 Processor -- start 2023-07-12T15:27:47.493 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.489+0000 7f0d95b30700 1 -- start start 2023-07-12T15:27:47.494 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.489+0000 7f0d95b30700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d900fe840 0x7f0d901080e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:47.494 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.489+0000 7f0d95b30700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d900ffa40 0x7f0d9019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:47.494 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.489+0000 7f0d95b30700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d901003d0 0x7f0d9019cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:47.494 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.489+0000 7f0d95b30700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9019d260 con 0x7f0d901003d0 2023-07-12T15:27:47.494 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.489+0000 7f0d95b30700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9019d3d0 con 0x7f0d900fe840 2023-07-12T15:27:47.494 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.489+0000 7f0d95b30700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9019d590 con 0x7f0d900ffa40 2023-07-12T15:27:47.494 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.489+0000 7f0d8f7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d900fe840 0x7f0d901080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:47.495 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.489+0000 7f0d8f7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d900fe840 0x7f0d901080e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44252/0 (socket says 172.21.15.145:44252) 2023-07-12T15:27:47.495 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.489+0000 7f0d8f7fe700 1 -- 172.21.15.145:0/598609331 learned_addr learned my addr 172.21.15.145:0/598609331 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:27:47.495 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.489+0000 7f0d8effd700 1 --2- 172.21.15.145:0/598609331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d900ffa40 0x7f0d9019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:47.495 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.489+0000 7f0d8ffff700 1 --2- 172.21.15.145:0/598609331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d901003d0 0x7f0d9019cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:47.495 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.490+0000 7f0d8f7fe700 1 -- 172.21.15.145:0/598609331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d900ffa40 msgr2=0x7f0d9019a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:47.495 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.490+0000 7f0d8f7fe700 1 --2- 172.21.15.145:0/598609331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d900ffa40 0x7f0d9019a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:47.495 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.490+0000 7f0d8f7fe700 1 -- 172.21.15.145:0/598609331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d901003d0 msgr2=0x7f0d9019cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:47.496 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.490+0000 7f0d8f7fe700 1 --2- 172.21.15.145:0/598609331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d901003d0 0x7f0d9019cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:47.496 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.490+0000 7f0d8f7fe700 1 -- 172.21.15.145:0/598609331 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d88009d00 con 0x7f0d900fe840 2023-07-12T15:27:47.496 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.490+0000 7f0d8f7fe700 1 --2- 172.21.15.145:0/598609331 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d900fe840 0x7f0d901080e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0d8000e970 tx=0x7f0d8000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:47.496 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.490+0000 7f0d8cff9700 1 -- 172.21.15.145:0/598609331 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d80009790 con 0x7f0d900fe840 2023-07-12T15:27:47.496 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.490+0000 7f0d8cff9700 1 -- 172.21.15.145:0/598609331 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f0d8000c7e0 con 0x7f0d900fe840 2023-07-12T15:27:47.496 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.490+0000 7f0d95b30700 1 -- 172.21.15.145:0/598609331 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9019d870 con 0x7f0d900fe840 2023-07-12T15:27:47.496 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.490+0000 7f0d8cff9700 1 -- 172.21.15.145:0/598609331 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d80003b60 con 0x7f0d900fe840 2023-07-12T15:27:47.497 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.490+0000 7f0d95b30700 1 -- 172.21.15.145:0/598609331 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9019dd20 con 0x7f0d900fe840 2023-07-12T15:27:47.497 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.491+0000 7f0d95b30700 1 -- 172.21.15.145:0/598609331 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d90105de0 con 0x7f0d900fe840 2023-07-12T15:27:47.497 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.491+0000 7f0d8cff9700 1 -- 172.21.15.145:0/598609331 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 15) v1 ==== 89784+0+0 (secure 0 0 0) 0x7f0d80003cc0 con 0x7f0d900fe840 2023-07-12T15:27:47.497 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.492+0000 7f0d8cff9700 1 --2- 172.21.15.145:0/598609331 >> v2:172.21.15.69:6800/284433772 conn(0x7f0d7806c720 0x7f0d7806ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:47.497 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.492+0000 7f0d8cff9700 1 -- 172.21.15.145:0/598609331 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5942+0+0 (secure 0 0 0) 0x7f0d80014070 con 0x7f0d900fe840 2023-07-12T15:27:47.497 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.492+0000 7f0d8effd700 1 --2- 172.21.15.145:0/598609331 >> v2:172.21.15.69:6800/284433772 conn(0x7f0d7806c720 0x7f0d7806ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:47.498 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.492+0000 7f0d8effd700 1 --2- 172.21.15.145:0/598609331 >> v2:172.21.15.69:6800/284433772 conn(0x7f0d7806c720 0x7f0d7806ebd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0d90108660 tx=0x7f0d840073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:47.498 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.497+0000 7f0d8cff9700 1 -- 172.21.15.145:0/598609331 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0d80056140 con 0x7f0d900fe840 2023-07-12T15:27:47.718 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.717+0000 7f0d95b30700 1 -- 172.21.15.145:0/598609331 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.145", "placement": "1;smithi145=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d90060f40 con 0x7f0d7806c720 2023-07-12T15:27:47.725 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.724+0000 7f0d8cff9700 1 -- 172.21.15.145:0/598609331 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (secure 0 0 0) 0x7f0d90060f40 con 0x7f0d7806c720 2023-07-12T15:27:47.725 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled iscsi.datapool update... 2023-07-12T15:27:47.727 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.726+0000 7f0d95b30700 1 -- 172.21.15.145:0/598609331 >> v2:172.21.15.69:6800/284433772 conn(0x7f0d7806c720 msgr2=0x7f0d7806ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:47.728 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.726+0000 7f0d95b30700 1 --2- 172.21.15.145:0/598609331 >> v2:172.21.15.69:6800/284433772 conn(0x7f0d7806c720 0x7f0d7806ebd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0d90108660 tx=0x7f0d840073d0 comp rx=0 tx=0).stop 2023-07-12T15:27:47.728 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.726+0000 7f0d95b30700 1 -- 172.21.15.145:0/598609331 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d900fe840 msgr2=0x7f0d901080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:47.728 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.726+0000 7f0d95b30700 1 --2- 172.21.15.145:0/598609331 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d900fe840 0x7f0d901080e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0d8000e970 tx=0x7f0d8000ec80 comp rx=0 tx=0).stop 2023-07-12T15:27:47.728 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.726+0000 7f0d95b30700 1 -- 172.21.15.145:0/598609331 shutdown_connections 2023-07-12T15:27:47.728 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.726+0000 7f0d95b30700 1 --2- 172.21.15.145:0/598609331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d901003d0 0x7f0d9019cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:47.728 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.726+0000 7f0d95b30700 1 --2- 172.21.15.145:0/598609331 >> v2:172.21.15.69:6800/284433772 conn(0x7f0d7806c720 0x7f0d7806ebd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:47.728 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.726+0000 7f0d95b30700 1 --2- 172.21.15.145:0/598609331 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d900fe840 0x7f0d901080e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:47.729 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.726+0000 7f0d95b30700 1 --2- 172.21.15.145:0/598609331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d900ffa40 0x7f0d9019a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:47.729 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.726+0000 7f0d95b30700 1 -- 172.21.15.145:0/598609331 >> 172.21.15.145:0/598609331 conn(0x7f0d900f9e70 msgr2=0x7f0d90102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:47.729 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.726+0000 7f0d95b30700 1 -- 172.21.15.145:0/598609331 shutdown_connections 2023-07-12T15:27:47.729 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:47.726+0000 7f0d95b30700 1 -- 172.21.15.145:0/598609331 wait complete. 2023-07-12T15:27:48.198 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2023-07-12T15:27:48.199 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:27:48.199 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2023-07-12T15:27:48.230 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-07-12T15:27:48.231 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2023-07-12T15:27:48.260 DEBUG:teuthology.orchestra.run.smithi145:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@iscsi.iscsi.a.service 2023-07-12T15:27:48.303 INFO:tasks.cephadm:Adding prometheus.a on smithi145 2023-07-12T15:27:48.304 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch apply prometheus '1;smithi145=a' 2023-07-12T15:27:48.330 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:-- Logs begin at Wed 2023-07-12 15:04:57 UTC. -- 2023-07-12T15:27:48.417 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-07-12T15:27:48.418 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-07-12T15:27:48.418 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: osdmap e67: 8 total, 8 up, 8 in 2023-07-12T15:27:48.418 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.418 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.419 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.419 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:48.419 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: pgmap v179: 132 pgs: 3 creating+activating, 10 creating+peering, 102 active+clean, 17 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-07-12T15:27:48.419 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:48.419 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:48.419 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: Checking dashboard <-> RGW credentials 2023-07-12T15:27:48.419 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.420 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.145", "placement": "1;smithi145=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:48.420 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: Saving service iscsi.datapool spec with placement smithi145=iscsi.a;count:1 2023-07-12T15:27:48.420 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.420 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:48.420 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:48.420 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:48.420 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.421 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-12T15:27:48.421 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-07-12T15:27:48.421 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:48.421 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:48 smithi145 ceph-mon[95801]: Deploying daemon iscsi.iscsi.a on smithi145 2023-07-12T15:27:48.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-07-12T15:27:48.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-07-12T15:27:48.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: osdmap e67: 8 total, 8 up, 8 in 2023-07-12T15:27:48.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:48.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: pgmap v179: 132 pgs: 3 creating+activating, 10 creating+peering, 102 active+clean, 17 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-07-12T15:27:48.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:48.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:48.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: Checking dashboard <-> RGW credentials 2023-07-12T15:27:48.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.145", "placement": "1;smithi145=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:48.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: Saving service iscsi.datapool spec with placement smithi145=iscsi.a;count:1 2023-07-12T15:27:48.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:48.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:48.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:48.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-12T15:27:48.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-07-12T15:27:48.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:48.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[94241]: Deploying daemon iscsi.iscsi.a on smithi145 2023-07-12T15:27:48.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-07-12T15:27:48.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-07-12T15:27:48.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: osdmap e67: 8 total, 8 up, 8 in 2023-07-12T15:27:48.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:48.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: pgmap v179: 132 pgs: 3 creating+activating, 10 creating+peering, 102 active+clean, 17 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-07-12T15:27:48.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:48.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:48.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: Checking dashboard <-> RGW credentials 2023-07-12T15:27:48.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.145", "placement": "1;smithi145=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:48.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: Saving service iscsi.datapool spec with placement smithi145=iscsi.a;count:1 2023-07-12T15:27:48.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:48.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:48.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:48.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:48.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-07-12T15:27:48.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-07-12T15:27:48.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:48.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:48 smithi069 ceph-mon[102777]: Deploying daemon iscsi.iscsi.a on smithi145 2023-07-12T15:27:49.185 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:27:50.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:27:49 smithi145 podman[118680]: 2023-07-12T15:27:50.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:27:50 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a-tcmu[118759]: log file path now is '/var/log/tcmu-runner.log' 2023-07-12T15:27:50.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:27:50 smithi145 bash[118680]: cbf135bf2b2a06f6b32336859f8dc89684f2bf901346482e856a5dfbbbb65144 2023-07-12T15:27:50.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:27:50 smithi145 podman[118798]: 2023-07-12T15:27:50.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:27:50 smithi145 bash[118798]: 888a200dabd401354d62560913d3217d472763c19004be0764f39b038f05595e 2023-07-12T15:27:50.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:27:50 smithi145 systemd[1]: Started Ceph iscsi.iscsi.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:27:50.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:50 smithi145 ceph-mon[95801]: pgmap v180: 132 pgs: 3 creating+activating, 7 creating+peering, 122 active+clean; 580 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 3.9 KiB/s wr, 88 op/s 2023-07-12T15:27:50.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:50 smithi145 ceph-mon[95801]: osdmap e68: 8 total, 8 up, 8 in 2023-07-12T15:27:50.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:50 smithi069 ceph-mon[94241]: pgmap v180: 132 pgs: 3 creating+activating, 7 creating+peering, 122 active+clean; 580 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 3.9 KiB/s wr, 88 op/s 2023-07-12T15:27:50.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:50 smithi069 ceph-mon[94241]: osdmap e68: 8 total, 8 up, 8 in 2023-07-12T15:27:50.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:50 smithi069 ceph-mon[102777]: pgmap v180: 132 pgs: 3 creating+activating, 7 creating+peering, 122 active+clean; 580 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 3.9 KiB/s wr, 88 op/s 2023-07-12T15:27:50.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:50 smithi069 ceph-mon[102777]: osdmap e68: 8 total, 8 up, 8 in 2023-07-12T15:27:51.016 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.014+0000 7f5df1ae7700 1 -- 172.21.15.145:0/1088260981 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dec101db0 msgr2=0x7f5dec108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:51.017 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.014+0000 7f5df1ae7700 1 --2- 172.21.15.145:0/1088260981 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dec101db0 0x7f5dec108670 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5de400b0a0 tx=0x7f5de400b3b0 comp rx=0 tx=0).stop 2023-07-12T15:27:51.017 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.014+0000 7f5df1ae7700 1 -- 172.21.15.145:0/1088260981 shutdown_connections 2023-07-12T15:27:51.017 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.014+0000 7f5df1ae7700 1 --2- 172.21.15.145:0/1088260981 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dec101db0 0x7f5dec108670 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:51.017 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.014+0000 7f5df1ae7700 1 --2- 172.21.15.145:0/1088260981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dec0fe440 0x7f5dec0fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:51.017 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.014+0000 7f5df1ae7700 1 --2- 172.21.15.145:0/1088260981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dec101420 0x7f5dec101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:51.018 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.014+0000 7f5df1ae7700 1 -- 172.21.15.145:0/1088260981 >> 172.21.15.145:0/1088260981 conn(0x7f5dec0f9e50 msgr2=0x7f5dec0fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:51.018 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.014+0000 7f5df1ae7700 1 -- 172.21.15.145:0/1088260981 shutdown_connections 2023-07-12T15:27:51.018 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.015+0000 7f5df1ae7700 1 -- 172.21.15.145:0/1088260981 wait complete. 2023-07-12T15:27:51.021 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.015+0000 7f5df1ae7700 1 Processor -- start 2023-07-12T15:27:51.021 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.015+0000 7f5df1ae7700 1 -- start start 2023-07-12T15:27:51.021 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5df1ae7700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dec0fe440 0x7f5dec195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:51.021 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5df1ae7700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dec101420 0x7f5dec195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:51.022 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5df1ae7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dec101db0 0x7f5dec19cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:51.022 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5df1ae7700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec19d1e0 con 0x7f5dec0fe440 2023-07-12T15:27:51.022 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5df1ae7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec19d320 con 0x7f5dec101db0 2023-07-12T15:27:51.022 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5df1ae7700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec19d4b0 con 0x7f5dec101420 2023-07-12T15:27:51.023 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5debfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dec101db0 0x7f5dec19cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:51.023 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5deb7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dec0fe440 0x7f5dec195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:51.023 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5deb7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dec0fe440 0x7f5dec195650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.145:46482/0 (socket says 172.21.15.145:46482) 2023-07-12T15:27:51.024 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5debfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dec101db0 0x7f5dec19cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44324/0 (socket says 172.21.15.145:44324) 2023-07-12T15:27:51.024 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5deb7fe700 1 -- 172.21.15.145:0/3585389932 learned_addr learned my addr 172.21.15.145:0/3585389932 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:27:51.024 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5deaffd700 1 --2- 172.21.15.145:0/3585389932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dec101420 0x7f5dec195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:51.025 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5debfff700 1 -- 172.21.15.145:0/3585389932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dec101420 msgr2=0x7f5dec195b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:51.025 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5debfff700 1 --2- 172.21.15.145:0/3585389932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dec101420 0x7f5dec195b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:51.025 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5debfff700 1 -- 172.21.15.145:0/3585389932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dec0fe440 msgr2=0x7f5dec195650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:51.025 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5debfff700 1 --2- 172.21.15.145:0/3585389932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dec0fe440 0x7f5dec195650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:51.026 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5debfff700 1 -- 172.21.15.145:0/3585389932 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de4009d00 con 0x7f5dec101db0 2023-07-12T15:27:51.026 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5deb7fe700 1 --2- 172.21.15.145:0/3585389932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dec0fe440 0x7f5dec195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:27:51.026 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5debfff700 1 --2- 172.21.15.145:0/3585389932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dec101db0 0x7f5dec19cc10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5de400b070 tx=0x7f5de40078e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:51.027 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5de8ff9700 1 -- 172.21.15.145:0/3585389932 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de400e040 con 0x7f5dec101db0 2023-07-12T15:27:51.027 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5df1ae7700 1 -- 172.21.15.145:0/3585389932 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dec0ff2d0 con 0x7f5dec101db0 2023-07-12T15:27:51.027 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5de8ff9700 1 -- 172.21.15.145:0/3585389932 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f5de400be40 con 0x7f5dec101db0 2023-07-12T15:27:51.027 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.016+0000 7f5de8ff9700 1 -- 172.21.15.145:0/3585389932 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de4005040 con 0x7f5dec101db0 2023-07-12T15:27:51.028 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.017+0000 7f5df1ae7700 1 -- 172.21.15.145:0/3585389932 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dec0ff7d0 con 0x7f5dec101db0 2023-07-12T15:27:51.028 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.017+0000 7f5df1ae7700 1 -- 172.21.15.145:0/3585389932 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dec107780 con 0x7f5dec101db0 2023-07-12T15:27:51.028 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.018+0000 7f5de8ff9700 1 -- 172.21.15.145:0/3585389932 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 15) v1 ==== 89784+0+0 (secure 0 0 0) 0x7f5de401d470 con 0x7f5dec101db0 2023-07-12T15:27:51.028 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.018+0000 7f5de8ff9700 1 --2- 172.21.15.145:0/3585389932 >> v2:172.21.15.69:6800/284433772 conn(0x7f5dd406c720 0x7f5dd406ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:51.028 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.019+0000 7f5de8ff9700 1 -- 172.21.15.145:0/3585389932 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (secure 0 0 0) 0x7f5de408cd30 con 0x7f5dec101db0 2023-07-12T15:27:51.029 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.019+0000 7f5deb7fe700 1 --2- 172.21.15.145:0/3585389932 >> v2:172.21.15.69:6800/284433772 conn(0x7f5dd406c720 0x7f5dd406ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:51.029 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.019+0000 7f5deb7fe700 1 --2- 172.21.15.145:0/3585389932 >> v2:172.21.15.69:6800/284433772 conn(0x7f5dd406c720 0x7f5dd406ebd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc005950 tx=0x7f5ddc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:51.030 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.023+0000 7f5de8ff9700 1 -- 172.21.15.145:0/3585389932 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5de4057230 con 0x7f5dec101db0 2023-07-12T15:27:51.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:27:50 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug Started the configuration object watcher 2023-07-12T15:27:51.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:27:50 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug Checking for config object changes every 1s 2023-07-12T15:27:51.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:27:50 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug Processing osd blocklist entries for this node 2023-07-12T15:27:51.235 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.234+0000 7f5df1ae7700 1 -- 172.21.15.145:0/3585389932 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f5dec0ff490 con 0x7f5dd406c720 2023-07-12T15:27:51.243 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.241+0000 7f5de8ff9700 1 -- 172.21.15.145:0/3585389932 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f5dec0ff490 con 0x7f5dd406c720 2023-07-12T15:27:51.243 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled prometheus update... 2023-07-12T15:27:51.245 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.243+0000 7f5df1ae7700 1 -- 172.21.15.145:0/3585389932 >> v2:172.21.15.69:6800/284433772 conn(0x7f5dd406c720 msgr2=0x7f5dd406ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:51.245 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.243+0000 7f5df1ae7700 1 --2- 172.21.15.145:0/3585389932 >> v2:172.21.15.69:6800/284433772 conn(0x7f5dd406c720 0x7f5dd406ebd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc005950 tx=0x7f5ddc009260 comp rx=0 tx=0).stop 2023-07-12T15:27:51.246 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.243+0000 7f5df1ae7700 1 -- 172.21.15.145:0/3585389932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dec101db0 msgr2=0x7f5dec19cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:51.246 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.243+0000 7f5df1ae7700 1 --2- 172.21.15.145:0/3585389932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dec101db0 0x7f5dec19cc10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5de400b070 tx=0x7f5de40078e0 comp rx=0 tx=0).stop 2023-07-12T15:27:51.246 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.243+0000 7f5df1ae7700 1 -- 172.21.15.145:0/3585389932 shutdown_connections 2023-07-12T15:27:51.246 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.243+0000 7f5df1ae7700 1 --2- 172.21.15.145:0/3585389932 >> v2:172.21.15.69:6800/284433772 conn(0x7f5dd406c720 0x7f5dd406ebd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:51.246 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.243+0000 7f5df1ae7700 1 --2- 172.21.15.145:0/3585389932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dec101db0 0x7f5dec19cc10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:51.247 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.243+0000 7f5df1ae7700 1 --2- 172.21.15.145:0/3585389932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dec0fe440 0x7f5dec195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:51.247 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.243+0000 7f5df1ae7700 1 --2- 172.21.15.145:0/3585389932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dec101420 0x7f5dec195b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:51.247 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.243+0000 7f5df1ae7700 1 -- 172.21.15.145:0/3585389932 >> 172.21.15.145:0/3585389932 conn(0x7f5dec0f9e50 msgr2=0x7f5dec1048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:51.247 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.244+0000 7f5df1ae7700 1 -- 172.21.15.145:0/3585389932 shutdown_connections 2023-07-12T15:27:51.247 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:51.244+0000 7f5df1ae7700 1 -- 172.21.15.145:0/3585389932 wait complete. 2023-07-12T15:27:51.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:27:51 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug Reading the configuration object to update local LIO configuration 2023-07-12T15:27:51.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:27:51 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug Configuration does not have an entry for this host(smithi145.front.sepia.ceph.com) - nothing to define to LIO 2023-07-12T15:27:51.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:27:51 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2023-07-12T15:27:51.721 DEBUG:teuthology.orchestra.run.smithi145:prometheus.a> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@prometheus.a.service 2023-07-12T15:27:51.723 INFO:tasks.cephadm:Adding node-exporter.a on smithi069 2023-07-12T15:27:51.723 INFO:tasks.cephadm:Adding node-exporter.b on smithi145 2023-07-12T15:27:51.723 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch apply node-exporter '2;smithi069=a;smithi145=b' 2023-07-12T15:27:51.753 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:-- Logs begin at Wed 2023-07-12 15:04:57 UTC. -- 2023-07-12T15:27:52.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:51 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:51 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:51 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:51 smithi145 ceph-mon[95801]: Checking pool "datapool" exists for service iscsi.datapool 2023-07-12T15:27:52.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:51 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:51 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:52.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/48934949' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-12T15:27:52.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:51 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[94241]: Checking pool "datapool" exists for service iscsi.datapool 2023-07-12T15:27:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/48934949' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-12T15:27:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[102777]: Checking pool "datapool" exists for service iscsi.datapool 2023-07-12T15:27:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:27:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/48934949' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-12T15:27:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:51 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:52.303 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:27:53.076 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:52 smithi145 ceph-mon[95801]: from='client.24502 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:53.077 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:52 smithi145 ceph-mon[95801]: Saving service prometheus spec with placement smithi145=a;count:1 2023-07-12T15:27:53.077 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:52 smithi145 ceph-mon[95801]: pgmap v182: 132 pgs: 3 creating+activating, 129 active+clean; 581 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 5.0 KiB/s wr, 151 op/s 2023-07-12T15:27:53.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:52 smithi069 ceph-mon[102777]: from='client.24502 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:52 smithi069 ceph-mon[102777]: Saving service prometheus spec with placement smithi145=a;count:1 2023-07-12T15:27:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:52 smithi069 ceph-mon[102777]: pgmap v182: 132 pgs: 3 creating+activating, 129 active+clean; 581 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 5.0 KiB/s wr, 151 op/s 2023-07-12T15:27:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:52 smithi069 ceph-mon[94241]: from='client.24502 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:52 smithi069 ceph-mon[94241]: Saving service prometheus spec with placement smithi145=a;count:1 2023-07-12T15:27:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:52 smithi069 ceph-mon[94241]: pgmap v182: 132 pgs: 3 creating+activating, 129 active+clean; 581 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 5.0 KiB/s wr, 151 op/s 2023-07-12T15:27:53.894 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.892+0000 7f0c9be25700 1 -- 172.21.15.145:0/1327284213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c94103580 msgr2=0x7f0c94105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:53.894 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.892+0000 7f0c9be25700 1 --2- 172.21.15.145:0/1327284213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c94103580 0x7f0c94105960 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009990 tx=0x7f0c84009ca0 comp rx=0 tx=0).stop 2023-07-12T15:27:53.894 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.892+0000 7f0c9be25700 1 -- 172.21.15.145:0/1327284213 shutdown_connections 2023-07-12T15:27:53.895 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.892+0000 7f0c9be25700 1 --2- 172.21.15.145:0/1327284213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c94105ea0 0x7f0c941082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:53.895 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.892+0000 7f0c9be25700 1 --2- 172.21.15.145:0/1327284213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c94100c60 0x7f0c94103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:53.895 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.892+0000 7f0c9be25700 1 --2- 172.21.15.145:0/1327284213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c94103580 0x7f0c94105960 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:53.895 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.892+0000 7f0c9be25700 1 -- 172.21.15.145:0/1327284213 >> 172.21.15.145:0/1327284213 conn(0x7f0c940fa6a0 msgr2=0x7f0c940fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:53.895 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.892+0000 7f0c9be25700 1 -- 172.21.15.145:0/1327284213 shutdown_connections 2023-07-12T15:27:53.895 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.892+0000 7f0c9be25700 1 -- 172.21.15.145:0/1327284213 wait complete. 2023-07-12T15:27:53.897 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.893+0000 7f0c9be25700 1 Processor -- start 2023-07-12T15:27:53.897 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.893+0000 7f0c9be25700 1 -- start start 2023-07-12T15:27:53.897 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.893+0000 7f0c9be25700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c94100c60 0x7f0c941005c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:53.897 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.893+0000 7f0c9be25700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c94103580 0x7f0c940fec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:53.897 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.893+0000 7f0c9be25700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c94105ea0 0x7f0c940ff150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:53.898 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.893+0000 7f0c9be25700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940ff8d0 con 0x7f0c94100c60 2023-07-12T15:27:53.898 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.893+0000 7f0c9be25700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940ffa40 con 0x7f0c94105ea0 2023-07-12T15:27:53.898 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.893+0000 7f0c9be25700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9419f620 con 0x7f0c94103580 2023-07-12T15:27:53.898 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c9a3c2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c94105ea0 0x7f0c940ff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:53.898 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c9a3c2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c94105ea0 0x7f0c940ff150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44340/0 (socket says 172.21.15.145:44340) 2023-07-12T15:27:53.898 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c9a3c2700 1 -- 172.21.15.145:0/1878219647 learned_addr learned my addr 172.21.15.145:0/1878219647 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:27:53.898 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c99bc1700 1 --2- 172.21.15.145:0/1878219647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c94100c60 0x7f0c941005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:53.899 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c993c0700 1 --2- 172.21.15.145:0/1878219647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c94103580 0x7f0c940fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:53.899 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c9a3c2700 1 -- 172.21.15.145:0/1878219647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c94103580 msgr2=0x7f0c940fec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:53.899 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c9a3c2700 1 --2- 172.21.15.145:0/1878219647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c94103580 0x7f0c940fec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:53.899 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c9a3c2700 1 -- 172.21.15.145:0/1878219647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c94100c60 msgr2=0x7f0c941005c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:53.899 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c9a3c2700 1 --2- 172.21.15.145:0/1878219647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c94100c60 0x7f0c941005c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:53.899 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c9a3c2700 1 -- 172.21.15.145:0/1878219647 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c90009520 con 0x7f0c94105ea0 2023-07-12T15:27:53.899 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c9a3c2700 1 --2- 172.21.15.145:0/1878219647 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c94105ea0 0x7f0c940ff150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00ef90 tx=0x7f0c8c00c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:53.899 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c8affd700 1 -- 172.21.15.145:0/1878219647 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c018070 con 0x7f0c94105ea0 2023-07-12T15:27:53.900 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c9be25700 1 -- 172.21.15.145:0/1878219647 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c840095f0 con 0x7f0c94105ea0 2023-07-12T15:27:53.900 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c8affd700 1 -- 172.21.15.145:0/1878219647 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f0c8c00f040 con 0x7f0c94105ea0 2023-07-12T15:27:53.900 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c8affd700 1 -- 172.21.15.145:0/1878219647 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c003d50 con 0x7f0c94105ea0 2023-07-12T15:27:53.900 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.894+0000 7f0c9be25700 1 -- 172.21.15.145:0/1878219647 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9419fbc0 con 0x7f0c94105ea0 2023-07-12T15:27:53.900 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.895+0000 7f0c9be25700 1 -- 172.21.15.145:0/1878219647 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c940fc210 con 0x7f0c94105ea0 2023-07-12T15:27:53.901 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.896+0000 7f0c8affd700 1 -- 172.21.15.145:0/1878219647 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 89835+0+0 (secure 0 0 0) 0x7f0c8c00cb00 con 0x7f0c94105ea0 2023-07-12T15:27:53.901 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.896+0000 7f0c8affd700 1 --2- 172.21.15.145:0/1878219647 >> v2:172.21.15.69:6800/284433772 conn(0x7f0c8006c7b0 0x7f0c8006ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:53.901 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.897+0000 7f0c8affd700 1 -- 172.21.15.145:0/1878219647 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (secure 0 0 0) 0x7f0c8c08b800 con 0x7f0c94105ea0 2023-07-12T15:27:53.901 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.897+0000 7f0c99bc1700 1 --2- 172.21.15.145:0/1878219647 >> v2:172.21.15.69:6800/284433772 conn(0x7f0c8006c7b0 0x7f0c8006ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:53.902 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.897+0000 7f0c99bc1700 1 --2- 172.21.15.145:0/1878219647 >> v2:172.21.15.69:6800/284433772 conn(0x7f0c8006c7b0 0x7f0c8006ec60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009d80 tx=0x7f0c90009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:53.902 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:53.901+0000 7f0c8affd700 1 -- 172.21.15.145:0/1878219647 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0c8c055c00 con 0x7f0c94105ea0 2023-07-12T15:27:54.120 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.119+0000 7f0c9be25700 1 -- 172.21.15.145:0/1878219647 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi069=a;smithi145=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c9402cfc0 con 0x7f0c8006c7b0 2023-07-12T15:27:54.127 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.126+0000 7f0c8affd700 1 -- 172.21.15.145:0/1878219647 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f0c90016070 con 0x7f0c8006c7b0 2023-07-12T15:27:54.128 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled node-exporter update... 2023-07-12T15:27:54.130 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.128+0000 7f0c9be25700 1 -- 172.21.15.145:0/1878219647 >> v2:172.21.15.69:6800/284433772 conn(0x7f0c8006c7b0 msgr2=0x7f0c8006ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:54.130 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.128+0000 7f0c9be25700 1 --2- 172.21.15.145:0/1878219647 >> v2:172.21.15.69:6800/284433772 conn(0x7f0c8006c7b0 0x7f0c8006ec60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009d80 tx=0x7f0c90009260 comp rx=0 tx=0).stop 2023-07-12T15:27:54.130 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.128+0000 7f0c9be25700 1 -- 172.21.15.145:0/1878219647 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c94105ea0 msgr2=0x7f0c940ff150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:54.130 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.128+0000 7f0c9be25700 1 --2- 172.21.15.145:0/1878219647 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c94105ea0 0x7f0c940ff150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00ef90 tx=0x7f0c8c00c450 comp rx=0 tx=0).stop 2023-07-12T15:27:54.130 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.128+0000 7f0c9be25700 1 -- 172.21.15.145:0/1878219647 shutdown_connections 2023-07-12T15:27:54.130 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.128+0000 7f0c9be25700 1 --2- 172.21.15.145:0/1878219647 >> v2:172.21.15.69:6800/284433772 conn(0x7f0c8006c7b0 0x7f0c8006ec60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:54.131 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.128+0000 7f0c9be25700 1 --2- 172.21.15.145:0/1878219647 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c94105ea0 0x7f0c940ff150 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:54.131 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.128+0000 7f0c9be25700 1 --2- 172.21.15.145:0/1878219647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c94100c60 0x7f0c941005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:54.131 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.128+0000 7f0c9be25700 1 --2- 172.21.15.145:0/1878219647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c94103580 0x7f0c940fec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:54.131 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.128+0000 7f0c9be25700 1 -- 172.21.15.145:0/1878219647 >> 172.21.15.145:0/1878219647 conn(0x7f0c940fa6a0 msgr2=0x7f0c941018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:54.131 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.128+0000 7f0c9be25700 1 -- 172.21.15.145:0/1878219647 shutdown_connections 2023-07-12T15:27:54.131 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:54.128+0000 7f0c9be25700 1 -- 172.21.15.145:0/1878219647 wait complete. 2023-07-12T15:27:54.518 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:54 smithi145 ceph-mon[95801]: mgrmap e16: y(active, since 4m), standbys: x 2023-07-12T15:27:54.519 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:54 smithi145 ceph-mon[95801]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 4.9 KiB/s wr, 153 op/s 2023-07-12T15:27:54.519 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:54 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:54.519 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:54 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:54.615 DEBUG:teuthology.orchestra.run.smithi069:node-exporter.a> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@node-exporter.a.service 2023-07-12T15:27:54.617 DEBUG:teuthology.orchestra.run.smithi145:node-exporter.b> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@node-exporter.b.service 2023-07-12T15:27:54.619 INFO:tasks.cephadm:Adding alertmanager.a on smithi069 2023-07-12T15:27:54.620 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch apply alertmanager '1;smithi069=a' 2023-07-12T15:27:54.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:54 smithi069 ceph-mon[102777]: mgrmap e16: y(active, since 4m), standbys: x 2023-07-12T15:27:54.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:54 smithi069 ceph-mon[102777]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 4.9 KiB/s wr, 153 op/s 2023-07-12T15:27:54.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:54 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:54.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:54 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:54.640 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:54 smithi069 ceph-mon[94241]: mgrmap e16: y(active, since 4m), standbys: x 2023-07-12T15:27:54.640 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:54 smithi069 ceph-mon[94241]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 4.9 KiB/s wr, 153 op/s 2023-07-12T15:27:54.640 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:54.641 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:54 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:54.643 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:-- Logs begin at Wed 2023-07-12 15:05:12 UTC. -- 2023-07-12T15:27:54.650 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:-- Logs begin at Wed 2023-07-12 15:04:57 UTC. -- 2023-07-12T15:27:55.202 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:27:55.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:55 smithi145 ceph-mon[95801]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi069=a;smithi145=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:55 smithi145 ceph-mon[95801]: Saving service node-exporter spec with placement smithi069=a;smithi145=b;count:2 2023-07-12T15:27:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:55 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:55 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:55 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:55 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:55.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:55 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:55.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:55 smithi145 ceph-mon[95801]: Deploying daemon prometheus.a on smithi145 2023-07-12T15:27:55.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[102777]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi069=a;smithi145=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:55.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[102777]: Saving service node-exporter spec with placement smithi069=a;smithi145=b;count:2 2023-07-12T15:27:55.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:55.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:55.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:55.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:55.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:55.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[102777]: Deploying daemon prometheus.a on smithi145 2023-07-12T15:27:55.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[94241]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi069=a;smithi145=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:55.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[94241]: Saving service node-exporter spec with placement smithi069=a;smithi145=b;count:2 2023-07-12T15:27:55.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:55.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:55.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:27:55.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:27:55.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:55.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:55 smithi069 ceph-mon[94241]: Deploying daemon prometheus.a on smithi145 2023-07-12T15:27:56.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:56 smithi145 ceph-mon[95801]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.2 KiB/s wr, 136 op/s 2023-07-12T15:27:56.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:56 smithi069 ceph-mon[102777]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.2 KiB/s wr, 136 op/s 2023-07-12T15:27:56.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:56 smithi069 ceph-mon[94241]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.2 KiB/s wr, 136 op/s 2023-07-12T15:27:56.969 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.966+0000 7feb02a88700 1 -- 172.21.15.145:0/3149563037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feafc100c00 msgr2=0x7feafc1074c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:56.969 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.966+0000 7feb02a88700 1 --2- 172.21.15.145:0/3149563037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feafc100c00 0x7feafc1074c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feaf400b0a0 tx=0x7feaf400b3b0 comp rx=0 tx=0).stop 2023-07-12T15:27:56.969 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.966+0000 7feb02a88700 1 -- 172.21.15.145:0/3149563037 shutdown_connections 2023-07-12T15:27:56.970 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.966+0000 7feb02a88700 1 --2- 172.21.15.145:0/3149563037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feafc100c00 0x7feafc1074c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:56.970 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.966+0000 7feb02a88700 1 --2- 172.21.15.145:0/3149563037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feafc0ff070 0x7feafc0ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:56.970 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.966+0000 7feb02a88700 1 --2- 172.21.15.145:0/3149563037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feafc100270 0x7feafc1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:56.970 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.966+0000 7feb02a88700 1 -- 172.21.15.145:0/3149563037 >> 172.21.15.145:0/3149563037 conn(0x7feafc0fa6a0 msgr2=0x7feafc0fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:56.970 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.966+0000 7feb02a88700 1 -- 172.21.15.145:0/3149563037 shutdown_connections 2023-07-12T15:27:56.970 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.966+0000 7feb02a88700 1 -- 172.21.15.145:0/3149563037 wait complete. 2023-07-12T15:27:56.972 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.967+0000 7feb02a88700 1 Processor -- start 2023-07-12T15:27:56.972 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.967+0000 7feb02a88700 1 -- start start 2023-07-12T15:27:56.972 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.967+0000 7feb02a88700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feafc0ff070 0x7feafc069120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:56.972 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.967+0000 7feb02a88700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feafc100270 0x7feafc19a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:56.972 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.967+0000 7feb02a88700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feafc100c00 0x7feafc19c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.967+0000 7feb02a88700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feafc19d170 con 0x7feafc100270 2023-07-12T15:27:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.967+0000 7feb02a88700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feafc19d2e0 con 0x7feafc0ff070 2023-07-12T15:27:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.967+0000 7feb02a88700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feafc19d4a0 con 0x7feafc100c00 2023-07-12T15:27:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.967+0000 7feb00824700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feafc0ff070 0x7feafc069120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.967+0000 7feb00824700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feafc0ff070 0x7feafc069120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44402/0 (socket says 172.21.15.145:44402) 2023-07-12T15:27:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.967+0000 7feb00824700 1 -- 172.21.15.145:0/3843779126 learned_addr learned my addr 172.21.15.145:0/3843779126 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:27:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.967+0000 7feafbfff700 1 --2- 172.21.15.145:0/3843779126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feafc100270 0x7feafc19a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:56.974 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.967+0000 7feb01025700 1 --2- 172.21.15.145:0/3843779126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feafc100c00 0x7feafc19c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:56.974 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.968+0000 7feb00824700 1 -- 172.21.15.145:0/3843779126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feafc100c00 msgr2=0x7feafc19c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:56.974 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.968+0000 7feb00824700 1 --2- 172.21.15.145:0/3843779126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feafc100c00 0x7feafc19c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:56.974 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.968+0000 7feb00824700 1 -- 172.21.15.145:0/3843779126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feafc100270 msgr2=0x7feafc19a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:56.974 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.968+0000 7feb00824700 1 --2- 172.21.15.145:0/3843779126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feafc100270 0x7feafc19a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:56.974 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.968+0000 7feb00824700 1 -- 172.21.15.145:0/3843779126 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf4009d00 con 0x7feafc0ff070 2023-07-12T15:27:56.974 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.968+0000 7feafbfff700 1 --2- 172.21.15.145:0/3843779126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feafc100270 0x7feafc19a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:27:56.975 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.968+0000 7feb00824700 1 --2- 172.21.15.145:0/3843779126 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feafc0ff070 0x7feafc069120 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feaf000d980 tx=0x7feaf000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:56.975 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.968+0000 7feaf9ffb700 1 -- 172.21.15.145:0/3843779126 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf0004020 con 0x7feafc0ff070 2023-07-12T15:27:56.975 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.968+0000 7feaf9ffb700 1 -- 172.21.15.145:0/3843779126 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7feaf0009730 con 0x7feafc0ff070 2023-07-12T15:27:56.975 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.968+0000 7feaf9ffb700 1 -- 172.21.15.145:0/3843779126 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf0003a20 con 0x7feafc0ff070 2023-07-12T15:27:56.975 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.968+0000 7feb02a88700 1 -- 172.21.15.145:0/3843779126 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feafc19d780 con 0x7feafc0ff070 2023-07-12T15:27:56.975 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.968+0000 7feb02a88700 1 -- 172.21.15.145:0/3843779126 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feafc19dc30 con 0x7feafc0ff070 2023-07-12T15:27:56.975 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.969+0000 7feb02a88700 1 -- 172.21.15.145:0/3843779126 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feafc1065c0 con 0x7feafc0ff070 2023-07-12T15:27:56.976 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.970+0000 7feaf9ffb700 1 -- 172.21.15.145:0/3843779126 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 89835+0+0 (secure 0 0 0) 0x7feaf0004180 con 0x7feafc0ff070 2023-07-12T15:27:56.976 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.970+0000 7feaf9ffb700 1 --2- 172.21.15.145:0/3843779126 >> v2:172.21.15.69:6800/284433772 conn(0x7feaec06c7b0 0x7feaec06ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:27:56.977 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.970+0000 7feaf9ffb700 1 -- 172.21.15.145:0/3843779126 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (secure 0 0 0) 0x7feaf008b840 con 0x7feafc0ff070 2023-07-12T15:27:56.977 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.970+0000 7feafbfff700 1 --2- 172.21.15.145:0/3843779126 >> v2:172.21.15.69:6800/284433772 conn(0x7feaec06c7b0 0x7feaec06ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:27:56.977 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.971+0000 7feafbfff700 1 --2- 172.21.15.145:0/3843779126 >> v2:172.21.15.69:6800/284433772 conn(0x7feaec06c7b0 0x7feaec06ec60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feae8005950 tx=0x7feae80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:27:56.977 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:56.974+0000 7feaf9ffb700 1 -- 172.21.15.145:0/3843779126 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7feaf0055b40 con 0x7feafc0ff070 2023-07-12T15:27:57.214 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.211+0000 7feb02a88700 1 -- 172.21.15.145:0/3843779126 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi069=a", "target": ["mon-mgr", ""]}) v1 -- 0x7feafc060f40 con 0x7feaec06c7b0 2023-07-12T15:27:57.221 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.218+0000 7feaf9ffb700 1 -- 172.21.15.145:0/3843779126 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7feafc060f40 con 0x7feaec06c7b0 2023-07-12T15:27:57.221 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled alertmanager update... 2023-07-12T15:27:57.223 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.220+0000 7feb02a88700 1 -- 172.21.15.145:0/3843779126 >> v2:172.21.15.69:6800/284433772 conn(0x7feaec06c7b0 msgr2=0x7feaec06ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:57.223 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.220+0000 7feb02a88700 1 --2- 172.21.15.145:0/3843779126 >> v2:172.21.15.69:6800/284433772 conn(0x7feaec06c7b0 0x7feaec06ec60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feae8005950 tx=0x7feae80058e0 comp rx=0 tx=0).stop 2023-07-12T15:27:57.224 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.220+0000 7feb02a88700 1 -- 172.21.15.145:0/3843779126 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feafc0ff070 msgr2=0x7feafc069120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:27:57.224 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.220+0000 7feb02a88700 1 --2- 172.21.15.145:0/3843779126 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feafc0ff070 0x7feafc069120 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feaf000d980 tx=0x7feaf000dc90 comp rx=0 tx=0).stop 2023-07-12T15:27:57.224 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.220+0000 7feb02a88700 1 -- 172.21.15.145:0/3843779126 shutdown_connections 2023-07-12T15:27:57.224 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.220+0000 7feb02a88700 1 --2- 172.21.15.145:0/3843779126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feafc100c00 0x7feafc19c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:57.224 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.220+0000 7feb02a88700 1 --2- 172.21.15.145:0/3843779126 >> v2:172.21.15.69:6800/284433772 conn(0x7feaec06c7b0 0x7feaec06ec60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:57.224 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.220+0000 7feb02a88700 1 --2- 172.21.15.145:0/3843779126 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feafc0ff070 0x7feafc069120 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:57.225 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.220+0000 7feb02a88700 1 --2- 172.21.15.145:0/3843779126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feafc100270 0x7feafc19a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:27:57.225 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.220+0000 7feb02a88700 1 -- 172.21.15.145:0/3843779126 >> 172.21.15.145:0/3843779126 conn(0x7feafc0fa6a0 msgr2=0x7feafc103700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:27:57.225 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.221+0000 7feb02a88700 1 -- 172.21.15.145:0/3843779126 shutdown_connections 2023-07-12T15:27:57.225 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:27:57.221+0000 7feb02a88700 1 -- 172.21.15.145:0/3843779126 wait complete. 2023-07-12T15:27:57.777 DEBUG:teuthology.orchestra.run.smithi069:alertmanager.a> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@alertmanager.a.service 2023-07-12T15:27:57.780 INFO:tasks.cephadm:Adding grafana.a on smithi145 2023-07-12T15:27:57.781 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph orch apply grafana '1;smithi145=a' 2023-07-12T15:27:57.811 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:-- Logs begin at Wed 2023-07-12 15:05:12 UTC. -- 2023-07-12T15:27:58.346 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:27:58.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:58 smithi069 ceph-mon[94241]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi069=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:58.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:58 smithi069 ceph-mon[94241]: Saving service alertmanager spec with placement smithi069=a;count:1 2023-07-12T15:27:58.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:58 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:58.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:58 smithi069 ceph-mon[94241]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 113 op/s 2023-07-12T15:27:58.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:58 smithi069 ceph-mon[102777]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi069=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:58.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:58 smithi069 ceph-mon[102777]: Saving service alertmanager spec with placement smithi069=a;count:1 2023-07-12T15:27:58.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:58 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:58.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:58 smithi069 ceph-mon[102777]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 113 op/s 2023-07-12T15:27:58.573 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:58 smithi145 ceph-mon[95801]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi069=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:27:58.574 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:58 smithi145 ceph-mon[95801]: Saving service alertmanager spec with placement smithi069=a;count:1 2023-07-12T15:27:58.574 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:58 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:58.574 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:58 smithi145 ceph-mon[95801]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 113 op/s 2023-07-12T15:27:59.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:27:59 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:59.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:27:59 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:27:59.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:27:59 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:28:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:00 smithi145 ceph-mon[95801]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 1.9 KiB/s wr, 68 op/s 2023-07-12T15:28:00.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:00 smithi069 ceph-mon[94241]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 1.9 KiB/s wr, 68 op/s 2023-07-12T15:28:00.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:00 smithi069 ceph-mon[102777]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 1.9 KiB/s wr, 68 op/s 2023-07-12T15:28:01.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:28:00 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:28:01.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:01 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:01.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:01 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:01.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:01 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:02.228 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.224+0000 7f1c83511700 1 -- 172.21.15.145:0/2054621239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7c100c00 msgr2=0x7f1c7c1074c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:02.228 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.224+0000 7f1c83511700 1 --2- 172.21.15.145:0/2054621239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7c100c00 0x7f1c7c1074c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400b0a0 tx=0x7f1c7400b3b0 comp rx=0 tx=0).stop 2023-07-12T15:28:02.228 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.224+0000 7f1c83511700 1 -- 172.21.15.145:0/2054621239 shutdown_connections 2023-07-12T15:28:02.228 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.224+0000 7f1c83511700 1 --2- 172.21.15.145:0/2054621239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7c100c00 0x7f1c7c1074c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:02.229 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.224+0000 7f1c83511700 1 --2- 172.21.15.145:0/2054621239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c7c0ff070 0x7f1c7c0ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:02.229 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.224+0000 7f1c83511700 1 --2- 172.21.15.145:0/2054621239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c7c100270 0x7f1c7c1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:02.229 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.224+0000 7f1c83511700 1 -- 172.21.15.145:0/2054621239 >> 172.21.15.145:0/2054621239 conn(0x7f1c7c0fa6a0 msgr2=0x7f1c7c0fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:02.229 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.224+0000 7f1c83511700 1 -- 172.21.15.145:0/2054621239 shutdown_connections 2023-07-12T15:28:02.229 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.224+0000 7f1c83511700 1 -- 172.21.15.145:0/2054621239 wait complete. 2023-07-12T15:28:02.229 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c83511700 1 Processor -- start 2023-07-12T15:28:02.230 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c83511700 1 -- start start 2023-07-12T15:28:02.230 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c83511700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7c0ff070 0x7f1c7c069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:02.230 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c83511700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c7c100270 0x7f1c7c19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:02.231 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c83511700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c7c100c00 0x7f1c7c19c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:02.231 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c83511700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c19d160 con 0x7f1c7c100270 2023-07-12T15:28:02.232 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c83511700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c19d2d0 con 0x7f1c7c0ff070 2023-07-12T15:28:02.232 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c83511700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c19d490 con 0x7f1c7c100c00 2023-07-12T15:28:02.232 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c812ad700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7c0ff070 0x7f1c7c069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:02.232 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c812ad700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7c0ff070 0x7f1c7c069140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44426/0 (socket says 172.21.15.145:44426) 2023-07-12T15:28:02.233 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c812ad700 1 -- 172.21.15.145:0/4234101081 learned_addr learned my addr 172.21.15.145:0/4234101081 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:28:02.234 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c80aac700 1 --2- 172.21.15.145:0/4234101081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c7c100270 0x7f1c7c19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:02.234 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c81aae700 1 --2- 172.21.15.145:0/4234101081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c7c100c00 0x7f1c7c19c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:02.236 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c812ad700 1 -- 172.21.15.145:0/4234101081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c7c100c00 msgr2=0x7f1c7c19c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:02.237 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.225+0000 7f1c812ad700 1 --2- 172.21.15.145:0/4234101081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c7c100c00 0x7f1c7c19c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:02.237 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.226+0000 7f1c812ad700 1 -- 172.21.15.145:0/4234101081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c7c100270 msgr2=0x7f1c7c19a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:02.237 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.226+0000 7f1c812ad700 1 --2- 172.21.15.145:0/4234101081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c7c100270 0x7f1c7c19a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:02.237 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.226+0000 7f1c812ad700 1 -- 172.21.15.145:0/4234101081 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c74009d00 con 0x7f1c7c0ff070 2023-07-12T15:28:02.238 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.226+0000 7f1c812ad700 1 --2- 172.21.15.145:0/4234101081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7c0ff070 0x7f1c7c069140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c00d980 tx=0x7f1c6c00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:02.238 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.226+0000 7f1c727fc700 1 -- 172.21.15.145:0/4234101081 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c004020 con 0x7f1c7c0ff070 2023-07-12T15:28:02.238 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.226+0000 7f1c727fc700 1 -- 172.21.15.145:0/4234101081 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f1c6c00f690 con 0x7f1c7c0ff070 2023-07-12T15:28:02.238 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.226+0000 7f1c83511700 1 -- 172.21.15.145:0/4234101081 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7c19d770 con 0x7f1c7c0ff070 2023-07-12T15:28:02.239 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.226+0000 7f1c83511700 1 -- 172.21.15.145:0/4234101081 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7c19dcc0 con 0x7f1c7c0ff070 2023-07-12T15:28:02.239 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.226+0000 7f1c727fc700 1 -- 172.21.15.145:0/4234101081 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c003960 con 0x7f1c7c0ff070 2023-07-12T15:28:02.239 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.227+0000 7f1c83511700 1 -- 172.21.15.145:0/4234101081 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c60005110 con 0x7f1c7c0ff070 2023-07-12T15:28:02.239 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.227+0000 7f1c727fc700 1 -- 172.21.15.145:0/4234101081 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 89835+0+0 (secure 0 0 0) 0x7f1c6c00fa70 con 0x7f1c7c0ff070 2023-07-12T15:28:02.239 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.228+0000 7f1c727fc700 1 --2- 172.21.15.145:0/4234101081 >> v2:172.21.15.69:6800/284433772 conn(0x7f1c6806c7b0 0x7f1c6806ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:02.239 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.228+0000 7f1c727fc700 1 -- 172.21.15.145:0/4234101081 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (secure 0 0 0) 0x7f1c6c08b5c0 con 0x7f1c7c0ff070 2023-07-12T15:28:02.240 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.228+0000 7f1c80aac700 1 --2- 172.21.15.145:0/4234101081 >> v2:172.21.15.69:6800/284433772 conn(0x7f1c6806c7b0 0x7f1c6806ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:02.240 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.228+0000 7f1c80aac700 1 --2- 172.21.15.145:0/4234101081 >> v2:172.21.15.69:6800/284433772 conn(0x7f1c6806c7b0 0x7f1c6806ec60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0696a0 tx=0x7f1c7800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:02.240 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.232+0000 7f1c727fc700 1 -- 172.21.15.145:0/4234101081 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1c6c055af0 con 0x7f1c7c0ff070 2023-07-12T15:28:02.437 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.434+0000 7f1c83511700 1 -- 172.21.15.145:0/4234101081 --> v2:172.21.15.69:6800/284433772 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c60000bf0 con 0x7f1c6806c7b0 2023-07-12T15:28:02.444 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.442+0000 7f1c727fc700 1 -- 172.21.15.145:0/4234101081 <== mgr.14152 v2:172.21.15.69:6800/284433772 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f1c60000bf0 con 0x7f1c6806c7b0 2023-07-12T15:28:02.445 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled grafana update... 2023-07-12T15:28:02.446 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.444+0000 7f1c83511700 1 -- 172.21.15.145:0/4234101081 >> v2:172.21.15.69:6800/284433772 conn(0x7f1c6806c7b0 msgr2=0x7f1c6806ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:02.447 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.444+0000 7f1c83511700 1 --2- 172.21.15.145:0/4234101081 >> v2:172.21.15.69:6800/284433772 conn(0x7f1c6806c7b0 0x7f1c6806ec60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0696a0 tx=0x7f1c7800b410 comp rx=0 tx=0).stop 2023-07-12T15:28:02.447 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.444+0000 7f1c83511700 1 -- 172.21.15.145:0/4234101081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7c0ff070 msgr2=0x7f1c7c069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:02.447 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.444+0000 7f1c83511700 1 --2- 172.21.15.145:0/4234101081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7c0ff070 0x7f1c7c069140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c00d980 tx=0x7f1c6c00dc90 comp rx=0 tx=0).stop 2023-07-12T15:28:02.447 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.444+0000 7f1c83511700 1 -- 172.21.15.145:0/4234101081 shutdown_connections 2023-07-12T15:28:02.448 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.444+0000 7f1c83511700 1 --2- 172.21.15.145:0/4234101081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c7c100c00 0x7f1c7c19c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:02.448 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.444+0000 7f1c83511700 1 --2- 172.21.15.145:0/4234101081 >> v2:172.21.15.69:6800/284433772 conn(0x7f1c6806c7b0 0x7f1c6806ec60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:02.448 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.444+0000 7f1c83511700 1 --2- 172.21.15.145:0/4234101081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7c0ff070 0x7f1c7c069140 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:02.448 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.444+0000 7f1c83511700 1 --2- 172.21.15.145:0/4234101081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c7c100270 0x7f1c7c19a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:02.449 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.444+0000 7f1c83511700 1 -- 172.21.15.145:0/4234101081 >> 172.21.15.145:0/4234101081 conn(0x7f1c7c0fa6a0 msgr2=0x7f1c7c103700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:02.449 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.444+0000 7f1c83511700 1 -- 172.21.15.145:0/4234101081 shutdown_connections 2023-07-12T15:28:02.449 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:02.444+0000 7f1c83511700 1 -- 172.21.15.145:0/4234101081 wait complete. 2023-07-12T15:28:02.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:02 smithi145 ceph-mon[95801]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 1.6 KiB/s wr, 59 op/s 2023-07-12T15:28:02.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:02 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:28:02.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:02 smithi069 ceph-mon[94241]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 1.6 KiB/s wr, 59 op/s 2023-07-12T15:28:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:02 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:28:02.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:02 smithi069 ceph-mon[102777]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 1.6 KiB/s wr, 59 op/s 2023-07-12T15:28:02.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:02 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:28:03.033 DEBUG:teuthology.orchestra.run.smithi145:grafana.a> sudo journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@grafana.a.service 2023-07-12T15:28:03.035 INFO:tasks.cephadm:Setting up client nodes... 2023-07-12T15:28:03.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-12T15:28:03.066 INFO:journalctl@ceph.grafana.a.smithi145.stdout:-- Logs begin at Wed 2023-07-12 15:04:57 UTC. -- 2023-07-12T15:28:03.594 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:03.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:03 smithi145 ceph-mon[95801]: from='client.24520 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:28:03.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:03 smithi145 ceph-mon[95801]: Saving service grafana spec with placement smithi145=a;count:1 2023-07-12T15:28:03.797 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:03 smithi069 ceph-mon[102777]: from='client.24520 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:28:03.797 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:03 smithi069 ceph-mon[102777]: Saving service grafana spec with placement smithi145=a;count:1 2023-07-12T15:28:03.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:03 smithi069 ceph-mon[94241]: from='client.24520 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2023-07-12T15:28:03.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:03 smithi069 ceph-mon[94241]: Saving service grafana spec with placement smithi145=a;count:1 2023-07-12T15:28:04.821 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.819+0000 7fab0122d700 1 -- 172.21.15.69:0/2035172008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc0ffce0 msgr2=0x7faafc107590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:04.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.819+0000 7fab0122d700 1 --2- 172.21.15.69:0/2035172008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc0ffce0 0x7faafc107590 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faaf400b0a0 tx=0x7faaf400b3b0 comp rx=0 tx=0).stop 2023-07-12T15:28:04.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.819+0000 7fab0122d700 1 -- 172.21.15.69:0/2035172008 shutdown_connections 2023-07-12T15:28:04.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.819+0000 7fab0122d700 1 --2- 172.21.15.69:0/2035172008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc0ffce0 0x7faafc107590 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:04.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.819+0000 7fab0122d700 1 --2- 172.21.15.69:0/2035172008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc1006d0 0x7faafc100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:04.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.819+0000 7fab0122d700 1 --2- 172.21.15.69:0/2035172008 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc0ff330 0x7faafc0ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:04.822 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.819+0000 7fab0122d700 1 -- 172.21.15.69:0/2035172008 >> 172.21.15.69:0/2035172008 conn(0x7faafc0f9e70 msgr2=0x7faafc0fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:04.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.819+0000 7fab0122d700 1 -- 172.21.15.69:0/2035172008 shutdown_connections 2023-07-12T15:28:04.823 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.819+0000 7fab0122d700 1 -- 172.21.15.69:0/2035172008 wait complete. 2023-07-12T15:28:04.824 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7fab0122d700 1 Processor -- start 2023-07-12T15:28:04.824 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7fab0122d700 1 -- start start 2023-07-12T15:28:04.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7fab0122d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc0ff330 0x7faafc0fed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:04.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7fab0122d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc0ffce0 0x7faafc19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:04.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7fab0122d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc1006d0 0x7faafc19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:04.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7fab0122d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc19d240 con 0x7faafc0ffce0 2023-07-12T15:28:04.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7fab0122d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc19d3b0 con 0x7faafc0ff330 2023-07-12T15:28:04.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7fab0122d700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faafc19d570 con 0x7faafc1006d0 2023-07-12T15:28:04.825 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7faafa59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc0ffce0 0x7faafc19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:04.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7faafb59e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc1006d0 0x7faafc19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:04.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7faafb59e700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc1006d0 0x7faafc19caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34248/0 (socket says 172.21.15.69:34248) 2023-07-12T15:28:04.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7faafa59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc0ffce0 0x7faafc19a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40818/0 (socket says 172.21.15.69:40818) 2023-07-12T15:28:04.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7faafb59e700 1 -- 172.21.15.69:0/3254499019 learned_addr learned my addr 172.21.15.69:0/3254499019 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:04.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7faafad9d700 1 --2- 172.21.15.69:0/3254499019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc0ff330 0x7faafc0fed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:04.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7faafa59c700 1 -- 172.21.15.69:0/3254499019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc1006d0 msgr2=0x7faafc19caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:04.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7faafa59c700 1 --2- 172.21.15.69:0/3254499019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc1006d0 0x7faafc19caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:04.826 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7faafa59c700 1 -- 172.21.15.69:0/3254499019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc0ff330 msgr2=0x7faafc0fed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:04.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7faafa59c700 1 --2- 172.21.15.69:0/3254499019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc0ff330 0x7faafc0fed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:04.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7faafa59c700 1 -- 172.21.15.69:0/3254499019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec009520 con 0x7faafc0ffce0 2023-07-12T15:28:04.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.820+0000 7faafb59e700 1 --2- 172.21.15.69:0/3254499019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc1006d0 0x7faafc19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:28:04.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.821+0000 7faafa59c700 1 --2- 172.21.15.69:0/3254499019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc0ffce0 0x7faafc19a460 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faaf000b450 tx=0x7faaf000b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:04.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.821+0000 7faae3fff700 1 -- 172.21.15.69:0/3254499019 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0011830 con 0x7faafc0ffce0 2023-07-12T15:28:04.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.821+0000 7faae3fff700 1 -- 172.21.15.69:0/3254499019 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7faaf000be60 con 0x7faafc0ffce0 2023-07-12T15:28:04.827 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.821+0000 7faae3fff700 1 -- 172.21.15.69:0/3254499019 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0010c30 con 0x7faafc0ffce0 2023-07-12T15:28:04.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.821+0000 7fab0122d700 1 -- 172.21.15.69:0/3254499019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf4009d00 con 0x7faafc0ffce0 2023-07-12T15:28:04.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.821+0000 7fab0122d700 1 -- 172.21.15.69:0/3254499019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faafc19dbf0 con 0x7faafc0ffce0 2023-07-12T15:28:04.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.822+0000 7fab0122d700 1 -- 172.21.15.69:0/3254499019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faafc1066f0 con 0x7faafc0ffce0 2023-07-12T15:28:04.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.822+0000 7faae3fff700 1 -- 172.21.15.69:0/3254499019 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) v1 ==== 89835+0+0 (secure 0 0 0) 0x7faaf0011990 con 0x7faafc0ffce0 2023-07-12T15:28:04.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.823+0000 7faae3fff700 1 --2- 172.21.15.69:0/3254499019 >> v2:172.21.15.69:6800/284433772 conn(0x7faae406c7b0 0x7faae406ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:04.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.823+0000 7faae3fff700 1 -- 172.21.15.69:0/3254499019 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (secure 0 0 0) 0x7faaf008b270 con 0x7faafc0ffce0 2023-07-12T15:28:04.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.823+0000 7faafad9d700 1 --2- 172.21.15.69:0/3254499019 >> v2:172.21.15.69:6800/284433772 conn(0x7faae406c7b0 0x7faae406ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:04.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.823+0000 7faafad9d700 1 --2- 172.21.15.69:0/3254499019 >> v2:172.21.15.69:6800/284433772 conn(0x7faae406c7b0 0x7faae406ec60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faafc0683a0 tx=0x7faaec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:04.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:04.827+0000 7faae3fff700 1 -- 172.21.15.69:0/3254499019 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7faaf0055780 con 0x7faafc0ffce0 2023-07-12T15:28:04.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:04 smithi145 ceph-mon[95801]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 682 B/s wr, 20 op/s 2023-07-12T15:28:04.851 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:04 smithi145 systemd[1]: Starting Ceph prometheus.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:28:04.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:04 smithi069 ceph-mon[102777]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 682 B/s wr, 20 op/s 2023-07-12T15:28:04.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:04 smithi069 ceph-mon[94241]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 682 B/s wr, 20 op/s 2023-07-12T15:28:05.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.090+0000 7fab0122d700 1 -- 172.21.15.69:0/3254499019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7faafc02cfb0 con 0x7faafc0ffce0 2023-07-12T15:28:05.096 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.094+0000 7faae3fff700 1 -- 172.21.15.69:0/3254499019 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7faaf0058da0 con 0x7faafc0ffce0 2023-07-12T15:28:05.097 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2023-07-12T15:28:05.097 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCFxq5kGUmEBRAAVZqCg3m2qUkweE7kwUA/JQ== 2023-07-12T15:28:05.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.096+0000 7fab0122d700 1 -- 172.21.15.69:0/3254499019 >> v2:172.21.15.69:6800/284433772 conn(0x7faae406c7b0 msgr2=0x7faae406ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:05.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.096+0000 7fab0122d700 1 --2- 172.21.15.69:0/3254499019 >> v2:172.21.15.69:6800/284433772 conn(0x7faae406c7b0 0x7faae406ec60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faafc0683a0 tx=0x7faaec009260 comp rx=0 tx=0).stop 2023-07-12T15:28:05.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.096+0000 7fab0122d700 1 -- 172.21.15.69:0/3254499019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc0ffce0 msgr2=0x7faafc19a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:05.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.096+0000 7fab0122d700 1 --2- 172.21.15.69:0/3254499019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc0ffce0 0x7faafc19a460 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faaf000b450 tx=0x7faaf000b760 comp rx=0 tx=0).stop 2023-07-12T15:28:05.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.096+0000 7fab0122d700 1 -- 172.21.15.69:0/3254499019 shutdown_connections 2023-07-12T15:28:05.099 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.096+0000 7fab0122d700 1 --2- 172.21.15.69:0/3254499019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faafc1006d0 0x7faafc19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:05.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.096+0000 7fab0122d700 1 --2- 172.21.15.69:0/3254499019 >> v2:172.21.15.69:6800/284433772 conn(0x7faae406c7b0 0x7faae406ec60 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:05.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.096+0000 7fab0122d700 1 --2- 172.21.15.69:0/3254499019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc0ff330 0x7faafc0fed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:05.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.096+0000 7fab0122d700 1 --2- 172.21.15.69:0/3254499019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faafc0ffce0 0x7faafc19a460 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:05.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.096+0000 7fab0122d700 1 -- 172.21.15.69:0/3254499019 >> 172.21.15.69:0/3254499019 conn(0x7faafc0f9e70 msgr2=0x7faafc103830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:05.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.097+0000 7fab0122d700 1 -- 172.21.15.69:0/3254499019 shutdown_connections 2023-07-12T15:28:05.100 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:05.097+0000 7fab0122d700 1 -- 172.21.15.69:0/3254499019 wait complete. 2023-07-12T15:28:05.314 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 podman[120592]: 2023-07-12T15:28:05.538 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:28:05.538 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-07-12T15:28:05.538 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-12T15:28:05.578 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-12T15:28:05.601 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 bash[120592]: 6be4846a21af2492f8132f4a4ec99f7ca7b9579c99ce569e7141641cebddb7ab 2023-07-12T15:28:05.601 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.313Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2023-07-12T15:28:05.601 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.313Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2023-07-12T15:28:05.601 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.313Z caller=main.go:570 level=info host_details="(Linux 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 x86_64 smithi145 (none))" 2023-07-12T15:28:05.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.313Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2023-07-12T15:28:05.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.313Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-12T15:28:05.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.315Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2023-07-12T15:28:05.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.315Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2023-07-12T15:28:05.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.317Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2023-07-12T15:28:05.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.317Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2023-07-12T15:28:05.603 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.318Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2023-07-12T15:28:05.603 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.318Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.55µs 2023-07-12T15:28:05.603 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.318Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2023-07-12T15:28:05.603 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.319Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2023-07-12T15:28:05.603 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.319Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.895µs wal_replay_duration=358.015µs wbl_replay_duration=134ns total_replay_duration=393.57µs 2023-07-12T15:28:05.603 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.320Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2023-07-12T15:28:05.603 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.320Z caller=main.go:1029 level=info msg="TSDB started" 2023-07-12T15:28:05.604 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.320Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-12T15:28:05.604 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.336Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.81583ms db_storage=660ns remote_storage=1.104µs web_handler=280ns query_engine=534ns scrape=265.394µs scrape_sd=56.315µs notify=736ns notify_sd=1.262µs rules=15.260769ms tracing=5.04µs 2023-07-12T15:28:05.604 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.336Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2023-07-12T15:28:05.604 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:28:05.336Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2023-07-12T15:28:05.604 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:28:05 smithi145 systemd[1]: Started Ceph prometheus.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:28:06.135 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.b/config 2023-07-12T15:28:06.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3254499019' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-12T15:28:06.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3254499019' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-12T15:28:06.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:28:06.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:28:06.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[102777]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2023-07-12T15:28:06.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:28:06.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-07-12T15:28:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3254499019' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-12T15:28:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3254499019' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-12T15:28:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:28:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:28:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[94241]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2023-07-12T15:28:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:28:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:05 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-07-12T15:28:06.347 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3254499019' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-12T15:28:06.347 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3254499019' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-12T15:28:06.347 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:28:06.347 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:28:06.347 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-mon[95801]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2023-07-12T15:28:06.347 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' 2023-07-12T15:28:06.348 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:05 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-07-12T15:28:06.600 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: ignoring --setuser ceph since I am not root 2023-07-12T15:28:06.601 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: ignoring --setgroup ceph since I am not root 2023-07-12T15:28:06.601 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:06 smithi145 ceph-mgr[97940]: -- 172.21.15.145:0/1032829695 <== mon.1 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55836a5cf1e0 con 0x55836a5a4800 2023-07-12T15:28:06.821 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ignoring --setuser ceph since I am not root 2023-07-12T15:28:06.822 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ignoring --setgroup ceph since I am not root 2023-07-12T15:28:06.822 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:06 smithi069 ceph-mgr[94512]: -- 172.21.15.69:0/3127104918 <== mon.2 v2:172.21.15.69:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559b7788f1e0 con 0x559b77864c00 2023-07-12T15:28:06.822 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:06.649+0000 7f40bdc85200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-12T15:28:06.822 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:06.729+0000 7f40bdc85200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-12T15:28:06.822 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:06.818+0000 7f40bdc85200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-12T15:28:06.932 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:06.662+0000 7f8e10dcf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-12T15:28:06.933 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:06.745+0000 7f8e10dcf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-12T15:28:06.933 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:06.835+0000 7f8e10dcf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-12T15:28:07.080 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:06.911+0000 7f40bdc85200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-12T15:28:07.300 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:06.931+0000 7f8e10dcf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-12T15:28:07.301 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:07 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:07.126+0000 7f8e10dcf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-12T15:28:07.508 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.506+0000 7f17ecc44700 1 -- 172.21.15.145:0/1177078502 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17e80ffdc0 msgr2=0x7f17e8100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:07.508 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.506+0000 7f17ecc44700 1 --2- 172.21.15.145:0/1177078502 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17e80ffdc0 0x7f17e8100230 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b0a0 tx=0x7f17e000b3b0 comp rx=0 tx=0).stop 2023-07-12T15:28:07.508 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.506+0000 7f17ecc44700 1 -- 172.21.15.145:0/1177078502 shutdown_connections 2023-07-12T15:28:07.508 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.506+0000 7f17ecc44700 1 --2- 172.21.15.145:0/1177078502 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17e80ffdc0 0x7f17e8100230 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:07.508 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.506+0000 7f17ecc44700 1 --2- 172.21.15.145:0/1177078502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17e80feac0 0x7f17e80feed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:07.509 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.506+0000 7f17ecc44700 1 --2- 172.21.15.145:0/1177078502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17e80ff410 0x7f17e80ff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:07.509 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.506+0000 7f17ecc44700 1 -- 172.21.15.145:0/1177078502 >> 172.21.15.145:0/1177078502 conn(0x7f17e80fa630 msgr2=0x7f17e80fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:07.509 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.506+0000 7f17ecc44700 1 -- 172.21.15.145:0/1177078502 shutdown_connections 2023-07-12T15:28:07.509 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.506+0000 7f17ecc44700 1 -- 172.21.15.145:0/1177078502 wait complete. 2023-07-12T15:28:07.511 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.506+0000 7f17ecc44700 1 Processor -- start 2023-07-12T15:28:07.511 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17ecc44700 1 -- start start 2023-07-12T15:28:07.511 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17ecc44700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17e80feac0 0x7f17e81978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:07.511 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17ecc44700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17e80ff410 0x7f17e8197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:07.512 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17ecc44700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17e80ffdc0 0x7f17e819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:07.512 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17ecc44700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e819f450 con 0x7f17e80feac0 2023-07-12T15:28:07.512 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17ecc44700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e819f590 con 0x7f17e80ff410 2023-07-12T15:28:07.512 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17ecc44700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e819f720 con 0x7f17e80ffdc0 2023-07-12T15:28:07.512 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17e5d9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17e80ff410 0x7f17e8197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:07.513 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17e5d9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17e80ff410 0x7f17e8197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44540/0 (socket says 172.21.15.145:44540) 2023-07-12T15:28:07.513 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17e5d9b700 1 -- 172.21.15.145:0/518287496 learned_addr learned my addr 172.21.15.145:0/518287496 (peer_addr_for_me v2:172.21.15.145:0/0) 2023-07-12T15:28:07.513 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17e659c700 1 --2- 172.21.15.145:0/518287496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17e80feac0 0x7f17e81978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:07.513 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17e6d9d700 1 --2- 172.21.15.145:0/518287496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17e80ffdc0 0x7f17e819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:07.513 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17e5d9b700 1 -- 172.21.15.145:0/518287496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17e80ffdc0 msgr2=0x7f17e819ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:07.513 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17e5d9b700 1 --2- 172.21.15.145:0/518287496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17e80ffdc0 0x7f17e819ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:07.513 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17e5d9b700 1 -- 172.21.15.145:0/518287496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17e80feac0 msgr2=0x7f17e81978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:07.514 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17e5d9b700 1 --2- 172.21.15.145:0/518287496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17e80feac0 0x7f17e81978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:07.514 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17e5d9b700 1 -- 172.21.15.145:0/518287496 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e0009d00 con 0x7f17e80ff410 2023-07-12T15:28:07.514 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17e5d9b700 1 --2- 172.21.15.145:0/518287496 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17e80ff410 0x7f17e8197e00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f17dc00e9a0 tx=0x7f17dc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:07.515 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.507+0000 7f17d77fe700 1 -- 172.21.15.145:0/518287496 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc009790 con 0x7f17e80ff410 2023-07-12T15:28:07.515 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.508+0000 7f17ecc44700 1 -- 172.21.15.145:0/518287496 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e819fa00 con 0x7f17e80ff410 2023-07-12T15:28:07.515 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.508+0000 7f17d77fe700 1 -- 172.21.15.145:0/518287496 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f17dc010430 con 0x7f17e80ff410 2023-07-12T15:28:07.515 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.508+0000 7f17d77fe700 1 -- 172.21.15.145:0/518287496 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc00cea0 con 0x7f17e80ff410 2023-07-12T15:28:07.515 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.508+0000 7f17ecc44700 1 -- 172.21.15.145:0/518287496 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e819ff00 con 0x7f17e80ff410 2023-07-12T15:28:07.516 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.509+0000 7f17ecc44700 1 -- 172.21.15.145:0/518287496 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e81098a0 con 0x7f17e80ff410 2023-07-12T15:28:07.516 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.509+0000 7f17d77fe700 1 -- 172.21.15.145:0/518287496 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 89849+0+0 (secure 0 0 0) 0x7f17dc00ca10 con 0x7f17e80ff410 2023-07-12T15:28:07.516 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.509+0000 7f17d77fe700 1 --2- 172.21.15.145:0/518287496 >> v2:172.21.15.69:6800/284433772 conn(0x7f17d006c800 0x7f17d006ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:07.516 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.509+0000 7f17e659c700 1 -- 172.21.15.145:0/518287496 >> v2:172.21.15.69:6800/284433772 conn(0x7f17d006c800 msgr2=0x7f17d006ecb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/284433772 2023-07-12T15:28:07.516 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.510+0000 7f17d77fe700 1 -- 172.21.15.145:0/518287496 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (secure 0 0 0) 0x7f17dc014070 con 0x7f17e80ff410 2023-07-12T15:28:07.516 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.510+0000 7f17e659c700 1 --2- 172.21.15.145:0/518287496 >> v2:172.21.15.69:6800/284433772 conn(0x7f17d006c800 0x7f17d006ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-12T15:28:07.517 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.514+0000 7f17d77fe700 1 -- 172.21.15.145:0/518287496 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f17dc051720 con 0x7f17e80ff410 2023-07-12T15:28:07.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:07 smithi145 ceph-mon[95801]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-07-12T15:28:07.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:07 smithi145 ceph-mon[95801]: mgrmap e17: y(active, since 4m), standbys: x 2023-07-12T15:28:07.601 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:07 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:07.446+0000 7f8e10dcf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-12T15:28:07.602 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:07 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:07.542+0000 7f8e10dcf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-12T15:28:07.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:07 smithi069 ceph-mon[102777]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-07-12T15:28:07.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:07 smithi069 ceph-mon[102777]: mgrmap e17: y(active, since 4m), standbys: x 2023-07-12T15:28:07.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:07 smithi069 ceph-mon[94241]: from='mgr.14152 172.21.15.69:0/3058456627' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-07-12T15:28:07.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:07 smithi069 ceph-mon[94241]: mgrmap e17: y(active, since 4m), standbys: x 2023-07-12T15:28:07.702 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:07 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:07.078+0000 7f40bdc85200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-12T15:28:07.703 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:07 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:07.385+0000 7f40bdc85200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-12T15:28:07.703 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:07 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:07.474+0000 7f40bdc85200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-12T15:28:07.703 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:07 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:07.605+0000 7f40bdc85200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-12T15:28:07.703 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:07 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:07.698+0000 7f40bdc85200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-12T15:28:07.712 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.711+0000 7f17e659c700 1 -- 172.21.15.145:0/518287496 >> v2:172.21.15.69:6800/284433772 conn(0x7f17d006c800 msgr2=0x7f17d006ecb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/284433772 2023-07-12T15:28:07.713 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.711+0000 7f17e659c700 1 --2- 172.21.15.145:0/518287496 >> v2:172.21.15.69:6800/284433772 conn(0x7f17d006c800 0x7f17d006ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-12T15:28:07.784 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.783+0000 7f17ecc44700 1 -- 172.21.15.145:0/518287496 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f17e802cc90 con 0x7f17e80ff410 2023-07-12T15:28:07.789 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.788+0000 7f17d77fe700 1 -- 172.21.15.145:0/518287496 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f17e802cc90 con 0x7f17e80ff410 2023-07-12T15:28:07.790 INFO:teuthology.orchestra.run.smithi145.stdout:[client.1] 2023-07-12T15:28:07.790 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQCHxq5k0efHLhAARbftg+1I36CW/7OjMqdKOw== 2023-07-12T15:28:07.792 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.790+0000 7f17ecc44700 1 -- 172.21.15.145:0/518287496 >> v2:172.21.15.69:6800/284433772 conn(0x7f17d006c800 msgr2=0x7f17d006ecb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-12T15:28:07.792 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.790+0000 7f17ecc44700 1 --2- 172.21.15.145:0/518287496 >> v2:172.21.15.69:6800/284433772 conn(0x7f17d006c800 0x7f17d006ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:07.792 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.790+0000 7f17ecc44700 1 -- 172.21.15.145:0/518287496 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17e80ff410 msgr2=0x7f17e8197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:07.792 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.790+0000 7f17ecc44700 1 --2- 172.21.15.145:0/518287496 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17e80ff410 0x7f17e8197e00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f17dc00e9a0 tx=0x7f17dc00ecb0 comp rx=0 tx=0).stop 2023-07-12T15:28:07.793 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.790+0000 7f17ecc44700 1 -- 172.21.15.145:0/518287496 shutdown_connections 2023-07-12T15:28:07.793 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.790+0000 7f17ecc44700 1 --2- 172.21.15.145:0/518287496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17e80ffdc0 0x7f17e819ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:07.793 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.790+0000 7f17ecc44700 1 --2- 172.21.15.145:0/518287496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17e80feac0 0x7f17e81978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:07.793 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.790+0000 7f17ecc44700 1 --2- 172.21.15.145:0/518287496 >> v2:172.21.15.69:6800/284433772 conn(0x7f17d006c800 0x7f17d006ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:07.793 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.790+0000 7f17ecc44700 1 --2- 172.21.15.145:0/518287496 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17e80ff410 0x7f17e8197e00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:07.793 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.790+0000 7f17ecc44700 1 -- 172.21.15.145:0/518287496 >> 172.21.15.145:0/518287496 conn(0x7f17e80fa630 msgr2=0x7f17e80fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:07.794 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.790+0000 7f17ecc44700 1 -- 172.21.15.145:0/518287496 shutdown_connections 2023-07-12T15:28:07.794 INFO:teuthology.orchestra.run.smithi145.stderr:2023-07-12T15:28:07.790+0000 7f17ecc44700 1 -- 172.21.15.145:0/518287496 wait complete. 2023-07-12T15:28:07.931 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:07 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:07.675+0000 7f8e10dcf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-12T15:28:07.931 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:07 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:07.772+0000 7f8e10dcf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-12T15:28:08.218 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-07-12T15:28:08.218 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2023-07-12T15:28:08.219 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-07-12T15:28:08.302 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-12T15:28:08.303 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-07-12T15:28:08.303 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph mgr dump --format=json 2023-07-12T15:28:08.483 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:08 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:08.117+0000 7f40bdc85200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-12T15:28:08.484 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:08 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:08.224+0000 7f8e10dcf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-12T15:28:08.735 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:08 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:08.701+0000 7f40bdc85200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-12T15:28:08.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.145:0/518287496' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-12T15:28:08.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:08 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-12T15:28:08.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:08 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-12T15:28:08.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.145:0/518287496' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-12T15:28:08.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:08 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-12T15:28:08.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:08 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-12T15:28:08.791 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.145:0/518287496' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-12T15:28:08.791 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:08 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-12T15:28:08.791 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:08 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-12T15:28:08.865 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:09.100 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:08 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:08.789+0000 7f8e10dcf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-12T15:28:09.101 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:08 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:08.960+0000 7f8e10dcf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-12T15:28:09.145 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:08 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:08.878+0000 7f40bdc85200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-12T15:28:09.976 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:09 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:09.709+0000 7f8e10dcf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-12T15:28:09.977 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:09 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:09.802+0000 7f8e10dcf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-12T15:28:09.978 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:09 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:09.701+0000 7f40bdc85200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-12T15:28:09.978 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:09 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:09.795+0000 7f40bdc85200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-12T15:28:10.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.074+0000 7fdd6e843700 1 -- 172.21.15.69:0/2400306031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd680fe6d0 msgr2=0x7fdd680feb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:10.081 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.074+0000 7fdd6e843700 1 --2- 172.21.15.69:0/2400306031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd680fe6d0 0x7fdd680feb40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009990 tx=0x7fdd50009ca0 comp rx=0 tx=0).stop 2023-07-12T15:28:10.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.074+0000 7fdd6e843700 1 -- 172.21.15.69:0/2400306031 shutdown_connections 2023-07-12T15:28:10.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.074+0000 7fdd6e843700 1 --2- 172.21.15.69:0/2400306031 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd680ff080 0x7fdd6810cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:10.082 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.074+0000 7fdd6e843700 1 --2- 172.21.15.69:0/2400306031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd68105130 0x7fdd68105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:10.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.074+0000 7fdd6e843700 1 --2- 172.21.15.69:0/2400306031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd680fe6d0 0x7fdd680feb40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:10.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.074+0000 7fdd6e843700 1 -- 172.21.15.69:0/2400306031 >> 172.21.15.69:0/2400306031 conn(0x7fdd68075740 msgr2=0x7fdd68075b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:10.083 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.074+0000 7fdd6e843700 1 -- 172.21.15.69:0/2400306031 shutdown_connections 2023-07-12T15:28:10.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.074+0000 7fdd6e843700 1 -- 172.21.15.69:0/2400306031 wait complete. 2023-07-12T15:28:10.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.075+0000 7fdd6e843700 1 Processor -- start 2023-07-12T15:28:10.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.075+0000 7fdd6e843700 1 -- start start 2023-07-12T15:28:10.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.075+0000 7fdd6e843700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd680fe6d0 0x7fdd680ff8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:10.084 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.075+0000 7fdd6e843700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd680ff080 0x7fdd680ffdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:10.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.075+0000 7fdd6e843700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd68105130 0x7fdd68100370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:10.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.075+0000 7fdd6e843700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd681a3e10 con 0x7fdd68105130 2023-07-12T15:28:10.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.075+0000 7fdd6e843700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd681a3f80 con 0x7fdd680ff080 2023-07-12T15:28:10.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.075+0000 7fdd6e843700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd681a4140 con 0x7fdd680fe6d0 2023-07-12T15:28:10.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.075+0000 7fdd67fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd680fe6d0 0x7fdd680ff8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:10.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.075+0000 7fdd67fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd680fe6d0 0x7fdd680ff8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34464/0 (socket says 172.21.15.69:34464) 2023-07-12T15:28:10.085 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.075+0000 7fdd6cde0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd68105130 0x7fdd68100370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:10.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.075+0000 7fdd67fff700 1 -- 172.21.15.69:0/878805658 learned_addr learned my addr 172.21.15.69:0/878805658 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:10.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.075+0000 7fdd677fe700 1 --2- 172.21.15.69:0/878805658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd680ff080 0x7fdd680ffdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:10.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.076+0000 7fdd6cde0700 1 -- 172.21.15.69:0/878805658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd680fe6d0 msgr2=0x7fdd680ff8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:10.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.076+0000 7fdd6cde0700 1 --2- 172.21.15.69:0/878805658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd680fe6d0 0x7fdd680ff8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:10.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.076+0000 7fdd6cde0700 1 -- 172.21.15.69:0/878805658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd680ff080 msgr2=0x7fdd680ffdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:10.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.076+0000 7fdd6cde0700 1 --2- 172.21.15.69:0/878805658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd680ff080 0x7fdd680ffdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:10.086 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.076+0000 7fdd6cde0700 1 -- 172.21.15.69:0/878805658 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd58009520 con 0x7fdd68105130 2023-07-12T15:28:10.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.076+0000 7fdd67fff700 1 --2- 172.21.15.69:0/878805658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd680fe6d0 0x7fdd680ff8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:28:10.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.076+0000 7fdd6cde0700 1 --2- 172.21.15.69:0/878805658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd68105130 0x7fdd68100370 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdd60007f10 tx=0x7fdd6000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:10.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.076+0000 7fdd657fa700 1 -- 172.21.15.69:0/878805658 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60009ea0 con 0x7fdd68105130 2023-07-12T15:28:10.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.076+0000 7fdd657fa700 1 -- 172.21.15.69:0/878805658 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fdd6000f040 con 0x7fdd68105130 2023-07-12T15:28:10.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.076+0000 7fdd6e843700 1 -- 172.21.15.69:0/878805658 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd500095f0 con 0x7fdd68105130 2023-07-12T15:28:10.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.076+0000 7fdd657fa700 1 -- 172.21.15.69:0/878805658 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60004260 con 0x7fdd68105130 2023-07-12T15:28:10.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.076+0000 7fdd6e843700 1 -- 172.21.15.69:0/878805658 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd681a4710 con 0x7fdd68105130 2023-07-12T15:28:10.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.077+0000 7fdd6e843700 1 -- 172.21.15.69:0/878805658 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6804e800 con 0x7fdd68105130 2023-07-12T15:28:10.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.078+0000 7fdd657fa700 1 -- 172.21.15.69:0/878805658 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) v1 ==== 89849+0+0 (secure 0 0 0) 0x7fdd6000db20 con 0x7fdd68105130 2023-07-12T15:28:10.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.078+0000 7fdd657fa700 1 --2- 172.21.15.69:0/878805658 >> v2:172.21.15.69:6800/284433772 conn(0x7fdd5406c800 0x7fdd5406ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:10.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.078+0000 7fdd67fff700 1 -- 172.21.15.69:0/878805658 >> v2:172.21.15.69:6800/284433772 conn(0x7fdd5406c800 msgr2=0x7fdd5406ecb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/284433772 2023-07-12T15:28:10.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.078+0000 7fdd657fa700 1 -- 172.21.15.69:0/878805658 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (secure 0 0 0) 0x7fdd6008b490 con 0x7fdd68105130 2023-07-12T15:28:10.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.078+0000 7fdd67fff700 1 --2- 172.21.15.69:0/878805658 >> v2:172.21.15.69:6800/284433772 conn(0x7fdd5406c800 0x7fdd5406ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-12T15:28:10.089 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.083+0000 7fdd657fa700 1 -- 172.21.15.69:0/878805658 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdd60055790 con 0x7fdd68105130 2023-07-12T15:28:10.258 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:09 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:09.974+0000 7f8e10dcf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-12T15:28:10.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.279+0000 7fdd67fff700 1 -- 172.21.15.69:0/878805658 >> v2:172.21.15.69:6800/284433772 conn(0x7fdd5406c800 msgr2=0x7fdd5406ecb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/284433772 2023-07-12T15:28:10.281 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.279+0000 7fdd67fff700 1 --2- 172.21.15.69:0/878805658 >> v2:172.21.15.69:6800/284433772 conn(0x7fdd5406c800 0x7fdd5406ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-12T15:28:10.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.335+0000 7fdd6e843700 1 -- 172.21.15.69:0/878805658 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fdd68100de0 con 0x7fdd68105130 2023-07-12T15:28:10.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.338+0000 7fdd657fa700 1 -- 172.21.15.69:0/878805658 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+172801 (secure 0 0 0) 0x7fdd60058db0 con 0x7fdd68105130 2023-07-12T15:28:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:28:10.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.341+0000 7fdd6e843700 1 -- 172.21.15.69:0/878805658 >> v2:172.21.15.69:6800/284433772 conn(0x7fdd5406c800 msgr2=0x7fdd5406ecb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-12T15:28:10.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.341+0000 7fdd6e843700 1 --2- 172.21.15.69:0/878805658 >> v2:172.21.15.69:6800/284433772 conn(0x7fdd5406c800 0x7fdd5406ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:10.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.341+0000 7fdd6e843700 1 -- 172.21.15.69:0/878805658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd68105130 msgr2=0x7fdd68100370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:10.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.341+0000 7fdd6e843700 1 --2- 172.21.15.69:0/878805658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd68105130 0x7fdd68100370 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdd60007f10 tx=0x7fdd6000d3b0 comp rx=0 tx=0).stop 2023-07-12T15:28:10.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.341+0000 7fdd6e843700 1 -- 172.21.15.69:0/878805658 shutdown_connections 2023-07-12T15:28:10.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.341+0000 7fdd6e843700 1 --2- 172.21.15.69:0/878805658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd68105130 0x7fdd68100370 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:10.348 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.341+0000 7fdd6e843700 1 --2- 172.21.15.69:0/878805658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd680fe6d0 0x7fdd680ff8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:10.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.341+0000 7fdd6e843700 1 --2- 172.21.15.69:0/878805658 >> v2:172.21.15.69:6800/284433772 conn(0x7fdd5406c800 0x7fdd5406ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:10.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.341+0000 7fdd6e843700 1 --2- 172.21.15.69:0/878805658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd680ff080 0x7fdd680ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:10.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.341+0000 7fdd6e843700 1 -- 172.21.15.69:0/878805658 >> 172.21.15.69:0/878805658 conn(0x7fdd68075740 msgr2=0x7fdd680fc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:10.349 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.341+0000 7fdd6e843700 1 -- 172.21.15.69:0/878805658 shutdown_connections 2023-07-12T15:28:10.350 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:10.341+0000 7fdd6e843700 1 -- 172.21.15.69:0/878805658 wait complete. 2023-07-12T15:28:10.600 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:10 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:10.256+0000 7f8e10dcf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-12T15:28:10.601 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:10 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:10.485+0000 7f8e10dcf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-12T15:28:10.665 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:09 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:09.975+0000 7f40bdc85200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-12T15:28:10.665 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:10 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:10.272+0000 7f40bdc85200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-12T15:28:10.665 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:10 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:10.511+0000 7f40bdc85200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-12T15:28:10.666 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:10 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:10.662+0000 7f40bdc85200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-12T15:28:10.822 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":17,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":284433772}]},"active_addr":"172.21.15.69:6800/284433772","active_change":"2023-07-12T15:23:43.423596+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.69:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":1778535974}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":2944961956}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":2814743090}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":2302771152}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":919046013}]}]} 2023-07-12T15:28:10.831 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-07-12T15:28:10.831 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-07-12T15:28:10.831 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd dump --format=json 2023-07-12T15:28:10.899 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:10 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:10.633+0000 7f8e10dcf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-12T15:28:11.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:10 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:10.948+0000 7f40bdc85200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-12T15:28:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/878805658' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-12T15:28:11.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/878805658' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-12T15:28:11.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/878805658' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-12T15:28:11.351 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:10 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:10.897+0000 7f8e10dcf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-12T15:28:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:11.850 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:11 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:11.597+0000 7f8e10dcf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-12T15:28:11.851 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:11 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:11.729+0000 7f8e10dcf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-12T15:28:11.955 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:11 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:11.702+0000 7f40bdc85200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-12T15:28:11.956 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:11 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:11.839+0000 7f40bdc85200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-12T15:28:12.236 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:12 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:12.040+0000 7f40bdc85200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-12T15:28:12.236 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:12 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:12.160+0000 7f40bdc85200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-12T15:28:12.258 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:11 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:11.922+0000 7f8e10dcf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-12T15:28:12.258 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:12 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:12.036+0000 7f8e10dcf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-12T15:28:12.600 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:12 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:12.256+0000 7f8e10dcf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-12T15:28:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.608+0000 7f143af93700 1 -- 172.21.15.69:0/3402871538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f143410cb20 msgr2=0x7f143410ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.608+0000 7f143af93700 1 --2- 172.21.15.69:0/3402871538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f143410cb20 0x7f143410ef00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f142c00b0a0 tx=0x7f142c00b3b0 comp rx=0 tx=0).stop 2023-07-12T15:28:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.608+0000 7f143af93700 1 -- 172.21.15.69:0/3402871538 shutdown_connections 2023-07-12T15:28:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.608+0000 7f143af93700 1 --2- 172.21.15.69:0/3402871538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f143410cb20 0x7f143410ef00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.608+0000 7f143af93700 1 --2- 172.21.15.69:0/3402871538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14340fef70 0x7f14340ff340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.608+0000 7f143af93700 1 --2- 172.21.15.69:0/3402871538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14340ff880 0x7f143410c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.608+0000 7f143af93700 1 -- 172.21.15.69:0/3402871538 >> 172.21.15.69:0/3402871538 conn(0x7f1434076160 msgr2=0x7f1434076560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:12.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.608+0000 7f143af93700 1 -- 172.21.15.69:0/3402871538 shutdown_connections 2023-07-12T15:28:12.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.608+0000 7f143af93700 1 -- 172.21.15.69:0/3402871538 wait complete. 2023-07-12T15:28:12.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.609+0000 7f143af93700 1 Processor -- start 2023-07-12T15:28:12.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.609+0000 7f143af93700 1 -- start start 2023-07-12T15:28:12.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.609+0000 7f143af93700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14340fef70 0x7f143419dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.609+0000 7f143af93700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14340ff880 0x7f143419e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.609+0000 7f143af93700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f143410cb20 0x7f1434197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.609+0000 7f143af93700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14341981c0 con 0x7f143410cb20 2023-07-12T15:28:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.609+0000 7f143af93700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1434198330 con 0x7f14340ff880 2023-07-12T15:28:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.609+0000 7f143af93700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14341984f0 con 0x7f14340fef70 2023-07-12T15:28:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.609+0000 7f1438d2f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14340fef70 0x7f143419dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:12.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.609+0000 7f1438d2f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14340fef70 0x7f143419dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34480/0 (socket says 172.21.15.69:34480) 2023-07-12T15:28:12.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.609+0000 7f1438d2f700 1 -- 172.21.15.69:0/1223122933 learned_addr learned my addr 172.21.15.69:0/1223122933 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:12.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.609+0000 7f1433fff700 1 --2- 172.21.15.69:0/1223122933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14340ff880 0x7f143419e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:12.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.609+0000 7f1439530700 1 --2- 172.21.15.69:0/1223122933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f143410cb20 0x7f1434197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:12.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.610+0000 7f1438d2f700 1 -- 172.21.15.69:0/1223122933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14340ff880 msgr2=0x7f143419e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:12.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.610+0000 7f1438d2f700 1 --2- 172.21.15.69:0/1223122933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14340ff880 0x7f143419e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:12.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.610+0000 7f1438d2f700 1 -- 172.21.15.69:0/1223122933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f143410cb20 msgr2=0x7f1434197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:12.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.610+0000 7f1438d2f700 1 --2- 172.21.15.69:0/1223122933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f143410cb20 0x7f1434197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:12.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.610+0000 7f1438d2f700 1 -- 172.21.15.69:0/1223122933 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1428009520 con 0x7f14340fef70 2023-07-12T15:28:12.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.610+0000 7f1438d2f700 1 --2- 172.21.15.69:0/1223122933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14340fef70 0x7f143419dad0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f142800eb30 tx=0x7f142800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:12.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.610+0000 7f1431ffb700 1 -- 172.21.15.69:0/1223122933 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1428009880 con 0x7f14340fef70 2023-07-12T15:28:12.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.610+0000 7f1431ffb700 1 -- 172.21.15.69:0/1223122933 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f14280044b0 con 0x7f14340fef70 2023-07-12T15:28:12.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.610+0000 7f1431ffb700 1 -- 172.21.15.69:0/1223122933 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1428003ca0 con 0x7f14340fef70 2023-07-12T15:28:12.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.610+0000 7f143af93700 1 -- 172.21.15.69:0/1223122933 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142c009d00 con 0x7f14340fef70 2023-07-12T15:28:12.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.610+0000 7f143af93700 1 -- 172.21.15.69:0/1223122933 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14341a48f0 con 0x7f14340fef70 2023-07-12T15:28:12.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.611+0000 7f143af93700 1 -- 172.21.15.69:0/1223122933 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143404e800 con 0x7f14340fef70 2023-07-12T15:28:12.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.612+0000 7f1431ffb700 1 -- 172.21.15.69:0/1223122933 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 17) v1 ==== 89849+0+0 (secure 0 0 0) 0x7f1428003e00 con 0x7f14340fef70 2023-07-12T15:28:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.612+0000 7f1431ffb700 1 --2- 172.21.15.69:0/1223122933 >> v2:172.21.15.69:6800/284433772 conn(0x7f142406c800 0x7f142406ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.612+0000 7f1433fff700 1 -- 172.21.15.69:0/1223122933 >> v2:172.21.15.69:6800/284433772 conn(0x7f142406c800 msgr2=0x7f142406ecb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/284433772 2023-07-12T15:28:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.612+0000 7f1431ffb700 1 -- 172.21.15.69:0/1223122933 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (secure 0 0 0) 0x7f1428014070 con 0x7f14340fef70 2023-07-12T15:28:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.612+0000 7f1433fff700 1 --2- 172.21.15.69:0/1223122933 >> v2:172.21.15.69:6800/284433772 conn(0x7f142406c800 0x7f142406ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-12T15:28:12.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.617+0000 7f1431ffb700 1 -- 172.21.15.69:0/1223122933 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1428056460 con 0x7f14340fef70 2023-07-12T15:28:12.735 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:12 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:12.409+0000 7f40bdc85200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-12T15:28:12.812 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.810+0000 7f143af93700 1 -- 172.21.15.69:0/1223122933 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14341a4b40 con 0x7f14340fef70 2023-07-12T15:28:12.814 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.811+0000 7f1431ffb700 1 -- 172.21.15.69:0/1223122933 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+20365 (secure 0 0 0) 0x7f1428059a80 con 0x7f14340fef70 2023-07-12T15:28:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:28:12.815 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":68,"fsid":"dc75a490-20c7-11ee-9b31-001a4aab830c","created":"2023-07-12T15:22:48.769414+0000","modified":"2023-07-12T15:27:49.946883+0000","last_up_change":"2023-07-12T15:27:26.534810+0000","last_in_change":"2023-07-12T15:27:10.083296+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-12T15:25:39.745272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2023-07-12T15:27:39.054435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-07-12T15:27:39.475716+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2023-07-12T15:27:40.408671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2023-07-12T15:27:42.377702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2023-07-12T15:27:44.459169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ca84d141-6fa9-46e6-9fa4-2b6ebe1d0cad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6801","nonce":657400327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":657400327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":657400327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":657400327}]},"public_addr":"172.21.15.69:6801/657400327","cluster_addr":"172.21.15.69:6802/657400327","heartbeat_back_addr":"172.21.15.69:6804/657400327","heartbeat_front_addr":"172.21.15.69:6803/657400327","state":["exists","up"]},{"osd":1,"uuid":"6e1e1865-07a1-4490-af3b-48f832fe203e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6805","nonce":2434701037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2434701037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2434701037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2434701037}]},"public_addr":"172.21.15.69:6805/2434701037","cluster_addr":"172.21.15.69:6806/2434701037","heartbeat_back_addr":"172.21.15.69:6808/2434701037","heartbeat_front_addr":"172.21.15.69:6807/2434701037","state":["exists","up"]},{"osd":2,"uuid":"67ce3cb4-9ddd-477c-a490-dd919b0cf1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6809","nonce":1319832966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1319832966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1319832966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1319832966}]},"public_addr":"172.21.15.69:6809/1319832966","cluster_addr":"172.21.15.69:6810/1319832966","heartbeat_back_addr":"172.21.15.69:6812/1319832966","heartbeat_front_addr":"172.21.15.69:6811/1319832966","state":["exists","up"]},{"osd":3,"uuid":"0a5e5d11-ad42-4663-8c6b-d04e9b968c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6813","nonce":2610582937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2610582937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2610582937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":2610582937}]},"public_addr":"172.21.15.69:6813/2610582937","cluster_addr":"172.21.15.69:6814/2610582937","heartbeat_back_addr":"172.21.15.69:6816/2610582937","heartbeat_front_addr":"172.21.15.69:6815/2610582937","state":["exists","up"]},{"osd":4,"uuid":"672b53e3-659e-408b-98e5-a3cbaf549b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":184109944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6801","nonce":184109944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":184109944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":184109944}]},"public_addr":"172.21.15.145:6800/184109944","cluster_addr":"172.21.15.145:6801/184109944","heartbeat_back_addr":"172.21.15.145:6803/184109944","heartbeat_front_addr":"172.21.15.145:6802/184109944","state":["exists","up"]},{"osd":5,"uuid":"975b3691-5232-4774-9b74-087ae97745a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":363217135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6805","nonce":363217135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":363217135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":363217135}]},"public_addr":"172.21.15.145:6804/363217135","cluster_addr":"172.21.15.145:6805/363217135","heartbeat_back_addr":"172.21.15.145:6807/363217135","heartbeat_front_addr":"172.21.15.145:6806/363217135","state":["exists","up"]},{"osd":6,"uuid":"2aa56dbc-dd4c-487c-8f74-00b05022b978","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":3456495522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6809","nonce":3456495522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":3456495522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":3456495522}]},"public_addr":"172.21.15.145:6808/3456495522","cluster_addr":"172.21.15.145:6809/3456495522","heartbeat_back_addr":"172.21.15.145:6811/3456495522","heartbeat_front_addr":"172.21.15.145:6810/3456495522","state":["exists","up"]},{"osd":7,"uuid":"55f492ca-7e77-4643-a798-b0899a6bf944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2623299546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6813","nonce":2623299546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6815","nonce":2623299546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2623299546}]},"public_addr":"172.21.15.145:6812/2623299546","cluster_addr":"172.21.15.145:6813/2623299546","heartbeat_back_addr":"172.21.15.145:6815/2623299546","heartbeat_front_addr":"172.21.15.145:6814/2623299546","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:24:55.893488+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:25:16.619394+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:25:37.932802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:25:59.145227+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:26:19.105589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:26:41.005568+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:27:03.198657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:27:24.827885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/3740256751":"2023-07-13T15:23:43.423483+0000","172.21.15.69:6800/339337620":"2023-07-13T15:23:43.423483+0000","172.21.15.69:0/3398378951":"2023-07-13T15:23:21.142752+0000","172.21.15.69:0/2429471300":"2023-07-13T15:23:43.423483+0000","172.21.15.69:0/2792247380":"2023-07-13T15:23:21.142752+0000","172.21.15.69:6800/654231506":"2023-07-13T15:23:21.142752+0000","172.21.15.69:0/1688014036":"2023-07-13T15:23:21.142752+0000","172.21.15.69:0/2361078360":"2023-07-13T15:23:43.423483+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-12T15:28:12.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.813+0000 7f1433fff700 1 -- 172.21.15.69:0/1223122933 >> v2:172.21.15.69:6800/284433772 conn(0x7f142406c800 msgr2=0x7f142406ecb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/284433772 2023-07-12T15:28:12.816 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.813+0000 7f1433fff700 1 --2- 172.21.15.69:0/1223122933 >> v2:172.21.15.69:6800/284433772 conn(0x7f142406c800 0x7f142406ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-12T15:28:12.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.814+0000 7f143af93700 1 -- 172.21.15.69:0/1223122933 >> v2:172.21.15.69:6800/284433772 conn(0x7f142406c800 msgr2=0x7f142406ecb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-12T15:28:12.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.814+0000 7f143af93700 1 --2- 172.21.15.69:0/1223122933 >> v2:172.21.15.69:6800/284433772 conn(0x7f142406c800 0x7f142406ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:12.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.814+0000 7f143af93700 1 -- 172.21.15.69:0/1223122933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14340fef70 msgr2=0x7f143419dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:12.817 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.814+0000 7f143af93700 1 --2- 172.21.15.69:0/1223122933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14340fef70 0x7f143419dad0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f142800eb30 tx=0x7f142800ee40 comp rx=0 tx=0).stop 2023-07-12T15:28:12.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.814+0000 7f143af93700 1 -- 172.21.15.69:0/1223122933 shutdown_connections 2023-07-12T15:28:12.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.814+0000 7f143af93700 1 --2- 172.21.15.69:0/1223122933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f143410cb20 0x7f1434197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:12.818 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.814+0000 7f143af93700 1 --2- 172.21.15.69:0/1223122933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14340fef70 0x7f143419dad0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:12.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.814+0000 7f143af93700 1 --2- 172.21.15.69:0/1223122933 >> v2:172.21.15.69:6800/284433772 conn(0x7f142406c800 0x7f142406ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:12.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.814+0000 7f143af93700 1 --2- 172.21.15.69:0/1223122933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14340ff880 0x7f143419e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:12.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.814+0000 7f143af93700 1 -- 172.21.15.69:0/1223122933 >> 172.21.15.69:0/1223122933 conn(0x7f1434076160 msgr2=0x7f14340fc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:12.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.814+0000 7f143af93700 1 -- 172.21.15.69:0/1223122933 shutdown_connections 2023-07-12T15:28:12.819 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:12.814+0000 7f143af93700 1 -- 172.21.15.69:0/1223122933 wait complete. 2023-07-12T15:28:13.118 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1223122933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-12T15:28:13.163 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1223122933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-12T15:28:13.163 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1223122933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-12T15:28:13.164 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-07-12T15:28:13.165 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd dump --format=json 2023-07-12T15:28:13.442 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:13.338+0000 7f40bdc85200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-12T15:28:13.442 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:28:13] ENGINE Bus STARTING 2023-07-12T15:28:13.477 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:13.116+0000 7f8e10dcf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-12T15:28:13.477 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: [12/Jul/2023:15:28:13] ENGINE Bus STARTING 2023-07-12T15:28:13.477 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: CherryPy Checker: 2023-07-12T15:28:13.477 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: The Application mounted at '' has an empty config. 2023-07-12T15:28:13.478 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: 2023-07-12T15:28:13.478 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: [12/Jul/2023:15:28:13] ENGINE Serving on http://:::9283 2023-07-12T15:28:13.478 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x[97916]: [12/Jul/2023:15:28:13] ENGINE Bus STARTED 2023-07-12T15:28:13.732 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:13.736 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: CherryPy Checker: 2023-07-12T15:28:13.736 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: The Application mounted at '' has an empty config. 2023-07-12T15:28:13.737 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:28:13.737 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:28:13] ENGINE Serving on http://:::9283 2023-07-12T15:28:13.737 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:28:13] ENGINE Bus STARTED 2023-07-12T15:28:14.164 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: Standby manager daemon x restarted 2023-07-12T15:28:14.165 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: Standby manager daemon x started 2023-07-12T15:28:14.165 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.? 172.21.15.145:0/551621667' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-12T15:28:14.165 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.? 172.21.15.145:0/551621667' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-12T15:28:14.165 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.? 172.21.15.145:0/551621667' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-12T15:28:14.165 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.? 172.21.15.145:0/551621667' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-12T15:28:14.165 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: Active manager daemon y restarted 2023-07-12T15:28:14.166 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: Activating manager daemon y 2023-07-12T15:28:14.166 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: osdmap e69: 8 total, 8 up, 8 in 2023-07-12T15:28:14.166 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: mgrmap e18: y(active, starting, since 0.00655479s), standbys: x 2023-07-12T15:28:14.166 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:28:14.166 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:28:14.166 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:28:14.166 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-12T15:28:14.167 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-12T15:28:14.167 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:28:14.167 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:28:14.167 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:28:14.167 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:28:14.167 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:28:14.167 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:28:14.168 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:28:14.168 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:28:14.168 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-12T15:28:14.168 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-12T15:28:14.168 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-12T15:28:14.168 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: Manager daemon y is now available 2023-07-12T15:28:14.168 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:28:14.168 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:28:14.169 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: Standby manager daemon x restarted 2023-07-12T15:28:14.169 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: Standby manager daemon x started 2023-07-12T15:28:14.169 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.? 172.21.15.145:0/551621667' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-12T15:28:14.169 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.? 172.21.15.145:0/551621667' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-12T15:28:14.169 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.? 172.21.15.145:0/551621667' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-12T15:28:14.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.? 172.21.15.145:0/551621667' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-12T15:28:14.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: Active manager daemon y restarted 2023-07-12T15:28:14.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: Activating manager daemon y 2023-07-12T15:28:14.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: osdmap e69: 8 total, 8 up, 8 in 2023-07-12T15:28:14.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: mgrmap e18: y(active, starting, since 0.00655479s), standbys: x 2023-07-12T15:28:14.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:28:14.170 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:28:14.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:28:14.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-12T15:28:14.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-12T15:28:14.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:28:14.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:28:14.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:28:14.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:28:14.171 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:28:14.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:28:14.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:28:14.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:28:14.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-12T15:28:14.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-12T15:28:14.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-12T15:28:14.172 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: Manager daemon y is now available 2023-07-12T15:28:14.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:28:14.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:28:14.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:14.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:28:14.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-12T15:28:14.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-12T15:28:14.173 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-12T15:28:14.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-12T15:28:14.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:14.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:28:14.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-12T15:28:14.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-12T15:28:14.174 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-12T15:28:14.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:13 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-12T15:28:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: Standby manager daemon x restarted 2023-07-12T15:28:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: Standby manager daemon x started 2023-07-12T15:28:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.? 172.21.15.145:0/551621667' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-12T15:28:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.? 172.21.15.145:0/551621667' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-12T15:28:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.? 172.21.15.145:0/551621667' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-12T15:28:14.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.? 172.21.15.145:0/551621667' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-12T15:28:14.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: Active manager daemon y restarted 2023-07-12T15:28:14.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: Activating manager daemon y 2023-07-12T15:28:14.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: osdmap e69: 8 total, 8 up, 8 in 2023-07-12T15:28:14.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: mgrmap e18: y(active, starting, since 0.00655479s), standbys: x 2023-07-12T15:28:14.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-12T15:28:14.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-12T15:28:14.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-12T15:28:14.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-12T15:28:14.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-12T15:28:14.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-12T15:28:14.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-12T15:28:14.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-12T15:28:14.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-12T15:28:14.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-12T15:28:14.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-12T15:28:14.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-12T15:28:14.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-12T15:28:14.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-12T15:28:14.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-12T15:28:14.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-12T15:28:14.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: Manager daemon y is now available 2023-07-12T15:28:14.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:28:14.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:28:14.359 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:14.359 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:28:14.359 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-12T15:28:14.360 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-12T15:28:14.360 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-12T15:28:14.360 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:13 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-12T15:28:14.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:28:14 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:28:15.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.162+0000 7fdf4221d700 1 -- 172.21.15.69:0/1885858946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf3c1054c0 msgr2=0x7fdf3c10d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:15.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.162+0000 7fdf4221d700 1 --2- 172.21.15.69:0/1885858946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf3c1054c0 0x7fdf3c10d040 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400b0d0 tx=0x7fdf3400b3e0 comp rx=0 tx=0).stop 2023-07-12T15:28:15.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.162+0000 7fdf4221d700 1 -- 172.21.15.69:0/1885858946 shutdown_connections 2023-07-12T15:28:15.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.162+0000 7fdf4221d700 1 --2- 172.21.15.69:0/1885858946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf3c1054c0 0x7fdf3c10d040 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:15.165 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.162+0000 7fdf4221d700 1 --2- 172.21.15.69:0/1885858946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf3c100470 0x7fdf3c100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:15.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.162+0000 7fdf4221d700 1 --2- 172.21.15.69:0/1885858946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf3c100e10 0x7fdf3c104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:15.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.162+0000 7fdf4221d700 1 -- 172.21.15.69:0/1885858946 >> 172.21.15.69:0/1885858946 conn(0x7fdf3c078190 msgr2=0x7fdf3c078590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:15.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.162+0000 7fdf4221d700 1 -- 172.21.15.69:0/1885858946 shutdown_connections 2023-07-12T15:28:15.166 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.162+0000 7fdf4221d700 1 -- 172.21.15.69:0/1885858946 wait complete. 2023-07-12T15:28:15.167 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.163+0000 7fdf4221d700 1 Processor -- start 2023-07-12T15:28:15.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.163+0000 7fdf4221d700 1 -- start start 2023-07-12T15:28:15.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.163+0000 7fdf4221d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf3c100470 0x7fdf3c1995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:15.168 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.163+0000 7fdf4221d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf3c100e10 0x7fdf3c199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:15.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.163+0000 7fdf4221d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf3c1054c0 0x7fdf3c193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:15.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.163+0000 7fdf4221d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c193ce0 con 0x7fdf3c1054c0 2023-07-12T15:28:15.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.163+0000 7fdf4221d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c193e50 con 0x7fdf3c100e10 2023-07-12T15:28:15.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.163+0000 7fdf4221d700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c194010 con 0x7fdf3c100470 2023-07-12T15:28:15.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.163+0000 7fdf3b7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf3c100470 0x7fdf3c1995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:15.169 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.163+0000 7fdf3bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf3c1054c0 0x7fdf3c193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:15.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.163+0000 7fdf3bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf3c1054c0 0x7fdf3c193670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41228/0 (socket says 172.21.15.69:41228) 2023-07-12T15:28:15.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.163+0000 7fdf3b7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf3c100470 0x7fdf3c1995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34652/0 (socket says 172.21.15.69:34652) 2023-07-12T15:28:15.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.163+0000 7fdf3bfff700 1 -- 172.21.15.69:0/2662664562 learned_addr learned my addr 172.21.15.69:0/2662664562 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:15.170 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.164+0000 7fdf3affd700 1 --2- 172.21.15.69:0/2662664562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf3c100e10 0x7fdf3c199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:15.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.164+0000 7fdf3bfff700 1 -- 172.21.15.69:0/2662664562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf3c100470 msgr2=0x7fdf3c1995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:15.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.164+0000 7fdf3bfff700 1 --2- 172.21.15.69:0/2662664562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf3c100470 0x7fdf3c1995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:15.171 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.164+0000 7fdf3bfff700 1 -- 172.21.15.69:0/2662664562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf3c100e10 msgr2=0x7fdf3c199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:15.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.164+0000 7fdf3bfff700 1 --2- 172.21.15.69:0/2662664562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf3c100e10 0x7fdf3c199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:15.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.164+0000 7fdf3bfff700 1 -- 172.21.15.69:0/2662664562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf34009d00 con 0x7fdf3c1054c0 2023-07-12T15:28:15.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.164+0000 7fdf3b7fe700 1 --2- 172.21.15.69:0/2662664562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf3c100470 0x7fdf3c1995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:28:15.172 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.164+0000 7fdf3bfff700 1 --2- 172.21.15.69:0/2662664562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf3c1054c0 0x7fdf3c193670 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400b770 tx=0x7fdf34008f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:15.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.164+0000 7fdf38ff9700 1 -- 172.21.15.69:0/2662664562 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3400bf30 con 0x7fdf3c1054c0 2023-07-12T15:28:15.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.164+0000 7fdf38ff9700 1 -- 172.21.15.69:0/2662664562 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fdf34014d60 con 0x7fdf3c1054c0 2023-07-12T15:28:15.173 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.164+0000 7fdf4221d700 1 -- 172.21.15.69:0/2662664562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3c194290 con 0x7fdf3c1054c0 2023-07-12T15:28:15.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.164+0000 7fdf38ff9700 1 -- 172.21.15.69:0/2662664562 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34025d30 con 0x7fdf3c1054c0 2023-07-12T15:28:15.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.164+0000 7fdf4221d700 1 -- 172.21.15.69:0/2662664562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3c1a05e0 con 0x7fdf3c1054c0 2023-07-12T15:28:15.174 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.165+0000 7fdf4221d700 1 -- 172.21.15.69:0/2662664562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3c04e800 con 0x7fdf3c1054c0 2023-07-12T15:28:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.166+0000 7fdf38ff9700 1 -- 172.21.15.69:0/2662664562 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 19) v1 ==== 89593+0+0 (secure 0 0 0) 0x7fdf34025440 con 0x7fdf3c1054c0 2023-07-12T15:28:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.166+0000 7fdf38ff9700 1 --2- 172.21.15.69:0/2662664562 >> v2:172.21.15.69:6800/1242716914 conn(0x7fdf2406c520 0x7fdf2406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.166+0000 7fdf38ff9700 1 -- 172.21.15.69:0/2662664562 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fdf34095b70 con 0x7fdf3c1054c0 2023-07-12T15:28:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.166+0000 7fdf3b7fe700 1 --2- 172.21.15.69:0/2662664562 >> v2:172.21.15.69:6800/1242716914 conn(0x7fdf2406c520 0x7fdf2406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.166+0000 7fdf3b7fe700 1 --2- 172.21.15.69:0/2662664562 >> v2:172.21.15.69:6800/1242716914 conn(0x7fdf2406c520 0x7fdf2406e9d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000f640 tx=0x7fdf30011040 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.170+0000 7fdf38ff9700 1 -- 172.21.15.69:0/2662664562 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdf340602c0 con 0x7fdf3c1054c0 2023-07-12T15:28:15.372 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.369+0000 7fdf4221d700 1 -- 172.21.15.69:0/2662664562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf3c194b80 con 0x7fdf3c1054c0 2023-07-12T15:28:15.373 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.371+0000 7fdf38ff9700 1 -- 172.21.15.69:0/2662664562 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+20695 (secure 0 0 0) 0x7fdf3402a020 con 0x7fdf3c1054c0 2023-07-12T15:28:15.374 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:28:15.374 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":69,"fsid":"dc75a490-20c7-11ee-9b31-001a4aab830c","created":"2023-07-12T15:22:48.769414+0000","modified":"2023-07-12T15:28:13.353883+0000","last_up_change":"2023-07-12T15:27:26.534810+0000","last_in_change":"2023-07-12T15:27:10.083296+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-12T15:25:39.745272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2023-07-12T15:27:39.054435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-07-12T15:27:39.475716+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2023-07-12T15:27:40.408671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2023-07-12T15:27:42.377702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2023-07-12T15:27:44.459169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ca84d141-6fa9-46e6-9fa4-2b6ebe1d0cad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6801","nonce":657400327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":657400327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":657400327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":657400327}]},"public_addr":"172.21.15.69:6801/657400327","cluster_addr":"172.21.15.69:6802/657400327","heartbeat_back_addr":"172.21.15.69:6804/657400327","heartbeat_front_addr":"172.21.15.69:6803/657400327","state":["exists","up"]},{"osd":1,"uuid":"6e1e1865-07a1-4490-af3b-48f832fe203e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6805","nonce":2434701037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2434701037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2434701037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":2434701037}]},"public_addr":"172.21.15.69:6805/2434701037","cluster_addr":"172.21.15.69:6806/2434701037","heartbeat_back_addr":"172.21.15.69:6808/2434701037","heartbeat_front_addr":"172.21.15.69:6807/2434701037","state":["exists","up"]},{"osd":2,"uuid":"67ce3cb4-9ddd-477c-a490-dd919b0cf1fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6809","nonce":1319832966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1319832966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1319832966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1319832966}]},"public_addr":"172.21.15.69:6809/1319832966","cluster_addr":"172.21.15.69:6810/1319832966","heartbeat_back_addr":"172.21.15.69:6812/1319832966","heartbeat_front_addr":"172.21.15.69:6811/1319832966","state":["exists","up"]},{"osd":3,"uuid":"0a5e5d11-ad42-4663-8c6b-d04e9b968c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6813","nonce":2610582937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2610582937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2610582937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":2610582937}]},"public_addr":"172.21.15.69:6813/2610582937","cluster_addr":"172.21.15.69:6814/2610582937","heartbeat_back_addr":"172.21.15.69:6816/2610582937","heartbeat_front_addr":"172.21.15.69:6815/2610582937","state":["exists","up"]},{"osd":4,"uuid":"672b53e3-659e-408b-98e5-a3cbaf549b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":184109944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6801","nonce":184109944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":184109944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":184109944}]},"public_addr":"172.21.15.145:6800/184109944","cluster_addr":"172.21.15.145:6801/184109944","heartbeat_back_addr":"172.21.15.145:6803/184109944","heartbeat_front_addr":"172.21.15.145:6802/184109944","state":["exists","up"]},{"osd":5,"uuid":"975b3691-5232-4774-9b74-087ae97745a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":363217135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6805","nonce":363217135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":363217135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":363217135}]},"public_addr":"172.21.15.145:6804/363217135","cluster_addr":"172.21.15.145:6805/363217135","heartbeat_back_addr":"172.21.15.145:6807/363217135","heartbeat_front_addr":"172.21.15.145:6806/363217135","state":["exists","up"]},{"osd":6,"uuid":"2aa56dbc-dd4c-487c-8f74-00b05022b978","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":3456495522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6809","nonce":3456495522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":3456495522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":3456495522}]},"public_addr":"172.21.15.145:6808/3456495522","cluster_addr":"172.21.15.145:6809/3456495522","heartbeat_back_addr":"172.21.15.145:6811/3456495522","heartbeat_front_addr":"172.21.15.145:6810/3456495522","state":["exists","up"]},{"osd":7,"uuid":"55f492ca-7e77-4643-a798-b0899a6bf944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2623299546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6813","nonce":2623299546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6815","nonce":2623299546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2623299546}]},"public_addr":"172.21.15.145:6812/2623299546","cluster_addr":"172.21.15.145:6813/2623299546","heartbeat_back_addr":"172.21.15.145:6815/2623299546","heartbeat_front_addr":"172.21.15.145:6814/2623299546","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:24:55.893488+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:25:16.619394+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:25:37.932802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:25:59.145227+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:26:19.105589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:26:41.005568+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:27:03.198657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-12T15:27:24.827885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/2944961956":"2023-07-13T15:28:13.353842+0000","172.21.15.69:6800/284433772":"2023-07-13T15:28:13.353842+0000","172.21.15.69:0/2302771152":"2023-07-13T15:28:13.353842+0000","172.21.15.69:0/1778535974":"2023-07-13T15:28:13.353842+0000","172.21.15.69:0/3740256751":"2023-07-13T15:23:43.423483+0000","172.21.15.69:6800/339337620":"2023-07-13T15:23:43.423483+0000","172.21.15.69:0/3398378951":"2023-07-13T15:23:21.142752+0000","172.21.15.69:0/919046013":"2023-07-13T15:28:13.353842+0000","172.21.15.69:0/2429471300":"2023-07-13T15:23:43.423483+0000","172.21.15.69:0/2792247380":"2023-07-13T15:23:21.142752+0000","172.21.15.69:6800/654231506":"2023-07-13T15:23:21.142752+0000","172.21.15.69:0/1688014036":"2023-07-13T15:23:21.142752+0000","172.21.15.69:0/2814743090":"2023-07-13T15:28:13.353842+0000","172.21.15.69:0/2361078360":"2023-07-13T15:23:43.423483+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-12T15:28:15.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.374+0000 7fdf4221d700 1 -- 172.21.15.69:0/2662664562 >> v2:172.21.15.69:6800/1242716914 conn(0x7fdf2406c520 msgr2=0x7fdf2406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:15.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.374+0000 7fdf4221d700 1 --2- 172.21.15.69:0/2662664562 >> v2:172.21.15.69:6800/1242716914 conn(0x7fdf2406c520 0x7fdf2406e9d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000f640 tx=0x7fdf30011040 comp rx=0 tx=0).stop 2023-07-12T15:28:15.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.375+0000 7fdf4221d700 1 -- 172.21.15.69:0/2662664562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf3c1054c0 msgr2=0x7fdf3c193670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:15.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.375+0000 7fdf4221d700 1 --2- 172.21.15.69:0/2662664562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf3c1054c0 0x7fdf3c193670 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400b770 tx=0x7fdf34008f20 comp rx=0 tx=0).stop 2023-07-12T15:28:15.377 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.375+0000 7fdf4221d700 1 -- 172.21.15.69:0/2662664562 shutdown_connections 2023-07-12T15:28:15.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.375+0000 7fdf4221d700 1 --2- 172.21.15.69:0/2662664562 >> v2:172.21.15.69:6800/1242716914 conn(0x7fdf2406c520 0x7fdf2406e9d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:15.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.375+0000 7fdf4221d700 1 --2- 172.21.15.69:0/2662664562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf3c1054c0 0x7fdf3c193670 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:15.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.375+0000 7fdf4221d700 1 --2- 172.21.15.69:0/2662664562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf3c100470 0x7fdf3c1995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:15.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.375+0000 7fdf4221d700 1 --2- 172.21.15.69:0/2662664562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf3c100e10 0x7fdf3c199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:15.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.375+0000 7fdf4221d700 1 -- 172.21.15.69:0/2662664562 >> 172.21.15.69:0/2662664562 conn(0x7fdf3c078190 msgr2=0x7fdf3c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:15.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.375+0000 7fdf4221d700 1 -- 172.21.15.69:0/2662664562 shutdown_connections 2023-07-12T15:28:15.378 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:15.375+0000 7fdf4221d700 1 -- 172.21.15.69:0/2662664562 wait complete. 2023-07-12T15:28:15.646 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[102777]: [12/Jul/2023:15:28:13] ENGINE Bus STARTING 2023-07-12T15:28:15.647 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[102777]: [12/Jul/2023:15:28:13] ENGINE Serving on https://172.21.15.69:7150 2023-07-12T15:28:15.647 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[102777]: [12/Jul/2023:15:28:13] ENGINE Serving on http://172.21.15.69:8765 2023-07-12T15:28:15.647 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[102777]: [12/Jul/2023:15:28:13] ENGINE Bus STARTED 2023-07-12T15:28:15.647 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[102777]: mgrmap e19: y(active, since 1.00932s), standbys: x 2023-07-12T15:28:15.648 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:15.648 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[102777]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:28:15.648 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[94241]: [12/Jul/2023:15:28:13] ENGINE Bus STARTING 2023-07-12T15:28:15.649 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[94241]: [12/Jul/2023:15:28:13] ENGINE Serving on https://172.21.15.69:7150 2023-07-12T15:28:15.649 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[94241]: [12/Jul/2023:15:28:13] ENGINE Serving on http://172.21.15.69:8765 2023-07-12T15:28:15.649 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[94241]: [12/Jul/2023:15:28:13] ENGINE Bus STARTED 2023-07-12T15:28:15.649 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[94241]: mgrmap e19: y(active, since 1.00932s), standbys: x 2023-07-12T15:28:15.649 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:15.649 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:15 smithi069 ceph-mon[94241]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:28:15.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:15 smithi145 ceph-mon[95801]: [12/Jul/2023:15:28:13] ENGINE Bus STARTING 2023-07-12T15:28:15.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:15 smithi145 ceph-mon[95801]: [12/Jul/2023:15:28:13] ENGINE Serving on https://172.21.15.69:7150 2023-07-12T15:28:15.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:15 smithi145 ceph-mon[95801]: [12/Jul/2023:15:28:13] ENGINE Serving on http://172.21.15.69:8765 2023-07-12T15:28:15.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:15 smithi145 ceph-mon[95801]: [12/Jul/2023:15:28:13] ENGINE Bus STARTED 2023-07-12T15:28:15.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:15 smithi145 ceph-mon[95801]: mgrmap e19: y(active, since 1.00932s), standbys: x 2023-07-12T15:28:15.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:15 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:15.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:15 smithi145 ceph-mon[95801]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:28:15.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2023-07-12T15:28:15.884 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2023-07-12T15:28:15.885 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2023-07-12T15:28:15.885 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2023-07-12T15:28:15.886 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2023-07-12T15:28:15.886 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2023-07-12T15:28:15.887 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2023-07-12T15:28:15.887 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2023-07-12T15:28:16.237 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:28:16] "GET /metrics HTTP/1.1" 200 35137 "" "Prometheus/2.43.0" 2023-07-12T15:28:16.690 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:16.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:16 smithi069 ceph-mon[102777]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:28:16.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2662664562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-12T15:28:16.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:16 smithi069 ceph-mon[94241]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:28:16.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2662664562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-12T15:28:16.850 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:16.850 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:16.851 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:16.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:16 smithi145 ceph-mon[95801]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:28:16.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2662664562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-12T15:28:16.852 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:16.892 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:17.071 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:17.071 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:17.664 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:17 smithi069 ceph-mon[102777]: mgrmap e20: y(active, since 3s), standbys: x 2023-07-12T15:28:17.665 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:17 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:17.665 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:17 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:17.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:17 smithi069 ceph-mon[94241]: mgrmap e20: y(active, since 3s), standbys: x 2023-07-12T15:28:17.666 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:17 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:17.667 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:17 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:17.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:17 smithi145 ceph-mon[95801]: mgrmap e20: y(active, since 3s), standbys: x 2023-07-12T15:28:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:17 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:17 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:18.640 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[102777]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:28:18.640 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:18.640 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:18.641 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.641 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.641 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.641 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.641 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.641 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.641 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.642 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.642 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:18.642 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[94241]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:28:18.642 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:18.642 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:18.642 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.643 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.643 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.643 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.643 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.643 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.643 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.644 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.644 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:18 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:18.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:18 smithi145 ceph-mon[95801]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:28:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:18 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:18 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:18 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:18 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:18 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:18 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:18 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:18 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:18 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:18 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:18.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:18 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:19.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:19 smithi069 ceph-mon[102777]: Adjusting osd_memory_target on smithi145 to 3780M 2023-07-12T15:28:19.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:19 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:19.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:19 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:19.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:19 smithi069 ceph-mon[94241]: Adjusting osd_memory_target on smithi145 to 3780M 2023-07-12T15:28:19.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:19 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:19.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:19 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:19.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:19 smithi145 ceph-mon[95801]: Adjusting osd_memory_target on smithi145 to 3780M 2023-07-12T15:28:19.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:19 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:19.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:19 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:20.198 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.196+0000 7fefcd34f700 1 -- 172.21.15.69:0/700938282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefc80ffca0 msgr2=0x7fefc8103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:20.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.196+0000 7fefcd34f700 1 --2- 172.21.15.69:0/700938282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefc80ffca0 0x7fefc8103c70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009920 tx=0x7fefb8009c30 comp rx=0 tx=0).stop 2023-07-12T15:28:20.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.196+0000 7fefcd34f700 1 -- 172.21.15.69:0/700938282 shutdown_connections 2023-07-12T15:28:20.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.196+0000 7fefcd34f700 1 --2- 172.21.15.69:0/700938282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fefc81043a0 0x7fefc810bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:20.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.196+0000 7fefcd34f700 1 --2- 172.21.15.69:0/700938282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fefc80ff300 0x7fefc80ff6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:20.199 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.196+0000 7fefcd34f700 1 --2- 172.21.15.69:0/700938282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefc80ffca0 0x7fefc8103c70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:20.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.196+0000 7fefcd34f700 1 -- 172.21.15.69:0/700938282 >> 172.21.15.69:0/700938282 conn(0x7fefc8076160 msgr2=0x7fefc8076560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:20.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.196+0000 7fefcd34f700 1 -- 172.21.15.69:0/700938282 shutdown_connections 2023-07-12T15:28:20.200 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.196+0000 7fefcd34f700 1 -- 172.21.15.69:0/700938282 wait complete. 2023-07-12T15:28:20.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.196+0000 7fefcd34f700 1 Processor -- start 2023-07-12T15:28:20.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefcd34f700 1 -- start start 2023-07-12T15:28:20.202 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefcd34f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fefc80ff300 0x7fefc819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:20.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefcd34f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fefc80ffca0 0x7fefc819aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:20.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefcd34f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefc81043a0 0x7fefc8194a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:20.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefcd34f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc8195090 con 0x7fefc81043a0 2023-07-12T15:28:20.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefcd34f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc8195200 con 0x7fefc80ff300 2023-07-12T15:28:20.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefcd34f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc81953c0 con 0x7fefc80ffca0 2023-07-12T15:28:20.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefc67fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fefc80ffca0 0x7fefc819aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:20.203 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefc77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefc81043a0 0x7fefc8194a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:20.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefc67fc700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fefc80ffca0 0x7fefc819aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34668/0 (socket says 172.21.15.69:34668) 2023-07-12T15:28:20.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefc67fc700 1 -- 172.21.15.69:0/1928939429 learned_addr learned my addr 172.21.15.69:0/1928939429 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:20.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefc6ffd700 1 --2- 172.21.15.69:0/1928939429 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fefc80ff300 0x7fefc819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:20.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefc67fc700 1 -- 172.21.15.69:0/1928939429 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fefc80ff300 msgr2=0x7fefc819a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:20.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefc67fc700 1 --2- 172.21.15.69:0/1928939429 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fefc80ff300 0x7fefc819a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:20.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefc67fc700 1 -- 172.21.15.69:0/1928939429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefc81043a0 msgr2=0x7fefc8194a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:20.204 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefc67fc700 1 --2- 172.21.15.69:0/1928939429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefc81043a0 0x7fefc8194a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:20.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefc67fc700 1 -- 172.21.15.69:0/1928939429 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc0009d00 con 0x7fefc80ffca0 2023-07-12T15:28:20.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefc6ffd700 1 --2- 172.21.15.69:0/1928939429 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fefc80ff300 0x7fefc819a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:28:20.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefc67fc700 1 --2- 172.21.15.69:0/1928939429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fefc80ffca0 0x7fefc819aee0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009230 tx=0x7fefb800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:20.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefaffff700 1 -- 172.21.15.69:0/1928939429 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb801c070 con 0x7fefc80ffca0 2023-07-12T15:28:20.205 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefcd34f700 1 -- 172.21.15.69:0/1928939429 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb8009520 con 0x7fefc80ffca0 2023-07-12T15:28:20.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefaffff700 1 -- 172.21.15.69:0/1928939429 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fefb8005040 con 0x7fefc80ffca0 2023-07-12T15:28:20.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.197+0000 7fefaffff700 1 -- 172.21.15.69:0/1928939429 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb8003880 con 0x7fefc80ffca0 2023-07-12T15:28:20.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.198+0000 7fefcd34f700 1 -- 172.21.15.69:0/1928939429 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc81a17c0 con 0x7fefc80ffca0 2023-07-12T15:28:20.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.198+0000 7fefcd34f700 1 -- 172.21.15.69:0/1928939429 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fefc810e960 con 0x7fefc80ffca0 2023-07-12T15:28:20.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.199+0000 7fefaffff700 1 -- 172.21.15.69:0/1928939429 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7fefb8020030 con 0x7fefc80ffca0 2023-07-12T15:28:20.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.199+0000 7fefaffff700 1 --2- 172.21.15.69:0/1928939429 >> v2:172.21.15.69:6800/1242716914 conn(0x7fefb006c810 0x7fefb006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:20.206 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.199+0000 7fefaffff700 1 -- 172.21.15.69:0/1928939429 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fefb808cb40 con 0x7fefc80ffca0 2023-07-12T15:28:20.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.199+0000 7fefc6ffd700 1 --2- 172.21.15.69:0/1928939429 >> v2:172.21.15.69:6800/1242716914 conn(0x7fefb006c810 0x7fefb006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:20.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.199+0000 7fefaffff700 1 --2- 172.21.15.69:0/1928939429 >> v2:172.21.15.69:6805/2434701037 conn(0x7fefb0075fe0 0x7fefb0078410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:20.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.199+0000 7fefaffff700 1 -- 172.21.15.69:0/1928939429 --> v2:172.21.15.69:6805/2434701037 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefb0078aa0 con 0x7fefb0075fe0 2023-07-12T15:28:20.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.199+0000 7fefaffff700 1 -- 172.21.15.69:0/1928939429 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefb80912c0 con 0x7fefc80ffca0 2023-07-12T15:28:20.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.200+0000 7fefc77fe700 1 --2- 172.21.15.69:0/1928939429 >> v2:172.21.15.69:6805/2434701037 conn(0x7fefb0075fe0 0x7fefb0078410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:20.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.200+0000 7fefc6ffd700 1 --2- 172.21.15.69:0/1928939429 >> v2:172.21.15.69:6800/1242716914 conn(0x7fefb006c810 0x7fefb006ecc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fefbc005950 tx=0x7fefbc0058e0 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:20.207 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.200+0000 7fefc77fe700 1 --2- 172.21.15.69:0/1928939429 >> v2:172.21.15.69:6805/2434701037 conn(0x7fefb0075fe0 0x7fefb0078410 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:20.208 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.201+0000 7fefaffff700 1 -- 172.21.15.69:0/1928939429 <== osd.1 v2:172.21.15.69:6805/2434701037 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fefb0078aa0 con 0x7fefb0075fe0 2023-07-12T15:28:20.218 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.216+0000 7fefcd34f700 1 -- 172.21.15.69:0/1928939429 --> v2:172.21.15.69:6805/2434701037 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fefc8061180 con 0x7fefb0075fe0 2023-07-12T15:28:20.219 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.217+0000 7fefaffff700 1 -- 172.21.15.69:0/1928939429 <== osd.1 v2:172.21.15.69:6805/2434701037 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fefc8061180 con 0x7fefb0075fe0 2023-07-12T15:28:20.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.217+0000 7fefcd34f700 1 -- 172.21.15.69:0/1928939429 >> v2:172.21.15.69:6805/2434701037 conn(0x7fefb0075fe0 msgr2=0x7fefb0078410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:20.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.217+0000 7fefcd34f700 1 --2- 172.21.15.69:0/1928939429 >> v2:172.21.15.69:6805/2434701037 conn(0x7fefb0075fe0 0x7fefb0078410 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:20.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.217+0000 7fefcd34f700 1 -- 172.21.15.69:0/1928939429 >> v2:172.21.15.69:6800/1242716914 conn(0x7fefb006c810 msgr2=0x7fefb006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:20.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.217+0000 7fefcd34f700 1 --2- 172.21.15.69:0/1928939429 >> v2:172.21.15.69:6800/1242716914 conn(0x7fefb006c810 0x7fefb006ecc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fefbc005950 tx=0x7fefbc0058e0 comp rx=0 tx=0).stop 2023-07-12T15:28:20.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.217+0000 7fefcd34f700 1 -- 172.21.15.69:0/1928939429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fefc80ffca0 msgr2=0x7fefc819aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:20.220 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.217+0000 7fefcd34f700 1 --2- 172.21.15.69:0/1928939429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fefc80ffca0 0x7fefc819aee0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009230 tx=0x7fefb800fa00 comp rx=0 tx=0).stop 2023-07-12T15:28:20.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.217+0000 7fefc77fe700 1 -- 172.21.15.69:0/1928939429 reap_dead start 2023-07-12T15:28:20.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.217+0000 7fefcd34f700 1 -- 172.21.15.69:0/1928939429 shutdown_connections 2023-07-12T15:28:20.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.217+0000 7fefcd34f700 1 -- 172.21.15.69:0/1928939429 >> 172.21.15.69:0/1928939429 conn(0x7fefc8076160 msgr2=0x7fefc80fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:20.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.217+0000 7fefcd34f700 1 -- 172.21.15.69:0/1928939429 shutdown_connections 2023-07-12T15:28:20.221 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.217+0000 7fefcd34f700 1 -- 172.21.15.69:0/1928939429 wait complete. 2023-07-12T15:28:20.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.473+0000 7fab0de16700 1 -- 172.21.15.69:0/3883119979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab080ff300 msgr2=0x7fab080ff6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:20.475 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.473+0000 7fab0de16700 1 --2- 172.21.15.69:0/3883119979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab080ff300 0x7fab080ff6d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7faaf8009920 tx=0x7faaf8009c30 comp rx=0 tx=0).stop 2023-07-12T15:28:20.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.473+0000 7fab0de16700 1 -- 172.21.15.69:0/3883119979 shutdown_connections 2023-07-12T15:28:20.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.473+0000 7fab0de16700 1 --2- 172.21.15.69:0/3883119979 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab081043a0 0x7fab0810bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:20.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.473+0000 7fab0de16700 1 --2- 172.21.15.69:0/3883119979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab080ff300 0x7fab080ff6d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:20.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.473+0000 7fab0de16700 1 --2- 172.21.15.69:0/3883119979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab080ffca0 0x7fab08103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:20.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.473+0000 7fab0de16700 1 -- 172.21.15.69:0/3883119979 >> 172.21.15.69:0/3883119979 conn(0x7fab08076160 msgr2=0x7fab08076560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:20.476 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.473+0000 7fab0de16700 1 -- 172.21.15.69:0/3883119979 shutdown_connections 2023-07-12T15:28:20.477 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.473+0000 7fab0de16700 1 -- 172.21.15.69:0/3883119979 wait complete. 2023-07-12T15:28:20.478 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.473+0000 7fab0de16700 1 Processor -- start 2023-07-12T15:28:20.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.473+0000 7fab0de16700 1 -- start start 2023-07-12T15:28:20.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab0de16700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab080ff300 0x7fab0819b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:20.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab0de16700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab080ffca0 0x7fab0819bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:20.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab0de16700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab081043a0 0x7fab08195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:20.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab0de16700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab08195ea0 con 0x7fab080ffca0 2023-07-12T15:28:20.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab0de16700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab08196010 con 0x7fab080ff300 2023-07-12T15:28:20.479 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab0de16700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab081961d0 con 0x7fab081043a0 2023-07-12T15:28:20.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab06ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab080ffca0 0x7fab0819bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:20.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab07fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab081043a0 0x7fab08195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:20.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab06ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab080ffca0 0x7fab0819bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41256/0 (socket says 172.21.15.69:41256) 2023-07-12T15:28:20.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab06ffd700 1 -- 172.21.15.69:0/3409022269 learned_addr learned my addr 172.21.15.69:0/3409022269 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:20.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab077fe700 1 --2- 172.21.15.69:0/3409022269 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab080ff300 0x7fab0819b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:20.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab06ffd700 1 -- 172.21.15.69:0/3409022269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab081043a0 msgr2=0x7fab08195830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:20.480 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab06ffd700 1 --2- 172.21.15.69:0/3409022269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab081043a0 0x7fab08195830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:20.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab06ffd700 1 -- 172.21.15.69:0/3409022269 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab080ff300 msgr2=0x7fab0819b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:20.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab06ffd700 1 --2- 172.21.15.69:0/3409022269 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab080ff300 0x7fab0819b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:20.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab06ffd700 1 -- 172.21.15.69:0/3409022269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc0095f0 con 0x7fab080ffca0 2023-07-12T15:28:20.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab07fff700 1 --2- 172.21.15.69:0/3409022269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab081043a0 0x7fab08195830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:28:20.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab06ffd700 1 --2- 172.21.15.69:0/3409022269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab080ffca0 0x7fab0819bcf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faafc00ec70 tx=0x7faafc00ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:20.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab04ff9700 1 -- 172.21.15.69:0/3409022269 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc0099d0 con 0x7fab080ffca0 2023-07-12T15:28:20.481 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab04ff9700 1 -- 172.21.15.69:0/3409022269 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7faafc0044b0 con 0x7fab080ffca0 2023-07-12T15:28:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab04ff9700 1 -- 172.21.15.69:0/3409022269 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc003e00 con 0x7fab080ffca0 2023-07-12T15:28:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab0de16700 1 -- 172.21.15.69:0/3409022269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf8009520 con 0x7fab080ffca0 2023-07-12T15:28:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.474+0000 7fab0de16700 1 -- 172.21.15.69:0/3409022269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab081a25d0 con 0x7fab080ffca0 2023-07-12T15:28:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.475+0000 7fab0de16700 1 -- 172.21.15.69:0/3409022269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fab0810f940 con 0x7fab080ffca0 2023-07-12T15:28:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.476+0000 7fab04ff9700 1 -- 172.21.15.69:0/3409022269 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7faafc003f60 con 0x7fab080ffca0 2023-07-12T15:28:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.476+0000 7fab04ff9700 1 --2- 172.21.15.69:0/3409022269 >> v2:172.21.15.69:6800/1242716914 conn(0x7faaf006c810 0x7faaf006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.476+0000 7fab04ff9700 1 -- 172.21.15.69:0/3409022269 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7faafc014070 con 0x7fab080ffca0 2023-07-12T15:28:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.476+0000 7fab077fe700 1 --2- 172.21.15.69:0/3409022269 >> v2:172.21.15.69:6800/1242716914 conn(0x7faaf006c810 0x7faaf006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:20.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.476+0000 7fab04ff9700 1 --2- 172.21.15.69:0/3409022269 >> v2:172.21.15.145:6804/363217135 conn(0x7faaf0076060 0x7faaf0078490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:20.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.476+0000 7fab04ff9700 1 -- 172.21.15.69:0/3409022269 --> v2:172.21.15.145:6804/363217135 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faaf0078b20 con 0x7faaf0076060 2023-07-12T15:28:20.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.476+0000 7fab04ff9700 1 -- 172.21.15.69:0/3409022269 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7faafc08c590 con 0x7fab080ffca0 2023-07-12T15:28:20.483 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.477+0000 7fab077fe700 1 --2- 172.21.15.69:0/3409022269 >> v2:172.21.15.69:6800/1242716914 conn(0x7faaf006c810 0x7faaf006ecc0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faaf8009810 tx=0x7faaf8019040 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:20.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.477+0000 7fab07fff700 1 --2- 172.21.15.69:0/3409022269 >> v2:172.21.15.145:6804/363217135 conn(0x7faaf0076060 0x7faaf0078490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:20.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.477+0000 7fab07fff700 1 --2- 172.21.15.69:0/3409022269 >> v2:172.21.15.145:6804/363217135 conn(0x7faaf0076060 0x7faaf0078490 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:20.484 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.478+0000 7fab04ff9700 1 -- 172.21.15.69:0/3409022269 <== osd.5 v2:172.21.15.145:6804/363217135 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7faaf0078b20 con 0x7faaf0076060 2023-07-12T15:28:20.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.493+0000 7fab0de16700 1 -- 172.21.15.69:0/3409022269 --> v2:172.21.15.145:6804/363217135 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fab0804e800 con 0x7faaf0076060 2023-07-12T15:28:20.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.494+0000 7fab04ff9700 1 -- 172.21.15.69:0/3409022269 <== osd.5 v2:172.21.15.145:6804/363217135 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fab0804e800 con 0x7faaf0076060 2023-07-12T15:28:20.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.494+0000 7fab0de16700 1 -- 172.21.15.69:0/3409022269 >> v2:172.21.15.145:6804/363217135 conn(0x7faaf0076060 msgr2=0x7faaf0078490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:20.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.494+0000 7fab0de16700 1 --2- 172.21.15.69:0/3409022269 >> v2:172.21.15.145:6804/363217135 conn(0x7faaf0076060 0x7faaf0078490 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:20.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.494+0000 7fab0de16700 1 -- 172.21.15.69:0/3409022269 >> v2:172.21.15.69:6800/1242716914 conn(0x7faaf006c810 msgr2=0x7faaf006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:20.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.494+0000 7fab0de16700 1 --2- 172.21.15.69:0/3409022269 >> v2:172.21.15.69:6800/1242716914 conn(0x7faaf006c810 0x7faaf006ecc0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faaf8009810 tx=0x7faaf8019040 comp rx=0 tx=0).stop 2023-07-12T15:28:20.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.494+0000 7fab0de16700 1 -- 172.21.15.69:0/3409022269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab080ffca0 msgr2=0x7fab0819bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:20.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.494+0000 7fab0de16700 1 --2- 172.21.15.69:0/3409022269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab080ffca0 0x7fab0819bcf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faafc00ec70 tx=0x7faafc00ef80 comp rx=0 tx=0).stop 2023-07-12T15:28:20.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.494+0000 7fab07fff700 1 -- 172.21.15.69:0/3409022269 reap_dead start 2023-07-12T15:28:20.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.495+0000 7fab0de16700 1 -- 172.21.15.69:0/3409022269 shutdown_connections 2023-07-12T15:28:20.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.495+0000 7fab0de16700 1 -- 172.21.15.69:0/3409022269 >> 172.21.15.69:0/3409022269 conn(0x7fab08076160 msgr2=0x7fab080fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:20.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.495+0000 7fab0de16700 1 -- 172.21.15.69:0/3409022269 shutdown_connections 2023-07-12T15:28:20.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.495+0000 7fab0de16700 1 -- 172.21.15.69:0/3409022269 wait complete. 2023-07-12T15:28:20.648 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:20 smithi069 ceph-mon[102777]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2023-07-12T15:28:20.649 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:20 smithi069 ceph-mon[94241]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2023-07-12T15:28:20.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:20 smithi145 ceph-mon[95801]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2023-07-12T15:28:22.001 INFO:teuthology.orchestra.run.smithi069.stdout:163208757269 2023-07-12T15:28:22.001 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.5 2023-07-12T15:28:22.004 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.002+0000 7fc797a79700 1 -- 172.21.15.69:0/3604112102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc790101610 msgr2=0x7fc79010c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.004 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.002+0000 7fc797a79700 1 --2- 172.21.15.69:0/3604112102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc790101610 0x7fc79010c000 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc78800b0a0 tx=0x7fc78800b3b0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.004 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.002+0000 7fc797a79700 1 -- 172.21.15.69:0/3604112102 shutdown_connections 2023-07-12T15:28:22.005 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.002+0000 7fc797a79700 1 --2- 172.21.15.69:0/3604112102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc790101610 0x7fc79010c000 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.005 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.002+0000 7fc797a79700 1 --2- 172.21.15.69:0/3604112102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7900ff400 0x7fc7900ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.005 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.002+0000 7fc797a79700 1 --2- 172.21.15.69:0/3604112102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc790100c60 0x7fc7901010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.005 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.002+0000 7fc797a79700 1 -- 172.21.15.69:0/3604112102 >> 172.21.15.69:0/3604112102 conn(0x7fc790076150 msgr2=0x7fc790076550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:22.005 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.002+0000 7fc797a79700 1 -- 172.21.15.69:0/3604112102 shutdown_connections 2023-07-12T15:28:22.006 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.002+0000 7fc797a79700 1 -- 172.21.15.69:0/3604112102 wait complete. 2023-07-12T15:28:22.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc797a79700 1 Processor -- start 2023-07-12T15:28:22.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc797a79700 1 -- start start 2023-07-12T15:28:22.007 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc797a79700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7900ff400 0x7fc790199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc797a79700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc790100c60 0x7fc790199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc797a79700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc790101610 0x7fc790193700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc797a79700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc790193d70 con 0x7fc790100c60 2023-07-12T15:28:22.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc797a79700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc790193ee0 con 0x7fc7900ff400 2023-07-12T15:28:22.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc797a79700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7901940a0 con 0x7fc790101610 2023-07-12T15:28:22.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc796016700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc790101610 0x7fc790193700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.008 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc795815700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7900ff400 0x7fc790199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc795815700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7900ff400 0x7fc790199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.69:48186/0 (socket says 172.21.15.69:48186) 2023-07-12T15:28:22.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc795815700 1 -- 172.21.15.69:0/3848102780 learned_addr learned my addr 172.21.15.69:0/3848102780 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:22.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc795815700 1 -- 172.21.15.69:0/3848102780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc790101610 msgr2=0x7fc790193700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc795815700 1 --2- 172.21.15.69:0/3848102780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc790101610 0x7fc790193700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc795815700 1 -- 172.21.15.69:0/3848102780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc790100c60 msgr2=0x7fc790199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc795014700 1 --2- 172.21.15.69:0/3848102780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc790100c60 0x7fc790199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.009 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.003+0000 7fc795815700 1 --2- 172.21.15.69:0/3848102780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc790100c60 0x7fc790199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.004+0000 7fc795815700 1 -- 172.21.15.69:0/3848102780 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc780009520 con 0x7fc7900ff400 2023-07-12T15:28:22.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.004+0000 7fc795014700 1 --2- 172.21.15.69:0/3848102780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc790100c60 0x7fc790199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-12T15:28:22.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.004+0000 7fc795815700 1 --2- 172.21.15.69:0/3848102780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7900ff400 0x7fc790199680 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00d950 tx=0x7fc78c00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:22.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.004+0000 7fc786ffd700 1 -- 172.21.15.69:0/3848102780 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c009940 con 0x7fc7900ff400 2023-07-12T15:28:22.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.004+0000 7fc786ffd700 1 -- 172.21.15.69:0/3848102780 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fc78c010460 con 0x7fc7900ff400 2023-07-12T15:28:22.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.004+0000 7fc797a79700 1 -- 172.21.15.69:0/3848102780 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc788009d00 con 0x7fc7900ff400 2023-07-12T15:28:22.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.004+0000 7fc786ffd700 1 -- 172.21.15.69:0/3848102780 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c00f620 con 0x7fc7900ff400 2023-07-12T15:28:22.010 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.004+0000 7fc797a79700 1 -- 172.21.15.69:0/3848102780 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc790194570 con 0x7fc7900ff400 2023-07-12T15:28:22.011 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.005+0000 7fc797a79700 1 -- 172.21.15.69:0/3848102780 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc79010da30 con 0x7fc7900ff400 2023-07-12T15:28:22.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.005+0000 7fc786ffd700 1 -- 172.21.15.69:0/3848102780 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7fc78c00f780 con 0x7fc7900ff400 2023-07-12T15:28:22.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.006+0000 7fc786ffd700 1 --2- 172.21.15.69:0/3848102780 >> v2:172.21.15.69:6800/1242716914 conn(0x7fc77c06c800 0x7fc77c06ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.006+0000 7fc786ffd700 1 -- 172.21.15.69:0/3848102780 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fc78c08b800 con 0x7fc7900ff400 2023-07-12T15:28:22.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.006+0000 7fc795014700 1 --2- 172.21.15.69:0/3848102780 >> v2:172.21.15.69:6800/1242716914 conn(0x7fc77c06c800 0x7fc77c06ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.006+0000 7fc786ffd700 1 --2- 172.21.15.69:0/3848102780 >> v2:172.21.15.69:6809/1319832966 conn(0x7fc77c075fd0 0x7fc77c078400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.006+0000 7fc786ffd700 1 -- 172.21.15.69:0/3848102780 --> v2:172.21.15.69:6809/1319832966 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc77c078a90 con 0x7fc77c075fd0 2023-07-12T15:28:22.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.006+0000 7fc796016700 1 --2- 172.21.15.69:0/3848102780 >> v2:172.21.15.69:6809/1319832966 conn(0x7fc77c075fd0 0x7fc77c078400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.006+0000 7fc786ffd700 1 -- 172.21.15.69:0/3848102780 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc78c059730 con 0x7fc7900ff400 2023-07-12T15:28:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.007+0000 7fc795014700 1 --2- 172.21.15.69:0/3848102780 >> v2:172.21.15.69:6800/1242716914 conn(0x7fc77c06c800 0x7fc77c06ecb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc790194b70 tx=0x7fc780009260 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.007+0000 7fc796016700 1 --2- 172.21.15.69:0/3848102780 >> v2:172.21.15.69:6809/1319832966 conn(0x7fc77c075fd0 0x7fc77c078400 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.008+0000 7fc786ffd700 1 -- 172.21.15.69:0/3848102780 <== osd.2 v2:172.21.15.69:6809/1319832966 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc77c078a90 con 0x7fc77c075fd0 2023-07-12T15:28:22.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.024+0000 7fc797a79700 1 -- 172.21.15.69:0/3848102780 --> v2:172.21.15.69:6809/1319832966 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc790194570 con 0x7fc77c075fd0 2023-07-12T15:28:22.027 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.024+0000 7fc786ffd700 1 -- 172.21.15.69:0/3848102780 <== osd.2 v2:172.21.15.69:6809/1319832966 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc790194570 con 0x7fc77c075fd0 2023-07-12T15:28:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.025+0000 7fc797a79700 1 -- 172.21.15.69:0/3848102780 >> v2:172.21.15.69:6809/1319832966 conn(0x7fc77c075fd0 msgr2=0x7fc77c078400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.028 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.025+0000 7fc797a79700 1 --2- 172.21.15.69:0/3848102780 >> v2:172.21.15.69:6809/1319832966 conn(0x7fc77c075fd0 0x7fc77c078400 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.025+0000 7fc797a79700 1 -- 172.21.15.69:0/3848102780 >> v2:172.21.15.69:6800/1242716914 conn(0x7fc77c06c800 msgr2=0x7fc77c06ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.025+0000 7fc797a79700 1 --2- 172.21.15.69:0/3848102780 >> v2:172.21.15.69:6800/1242716914 conn(0x7fc77c06c800 0x7fc77c06ecb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc790194b70 tx=0x7fc780009260 comp rx=0 tx=0).stop 2023-07-12T15:28:22.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.025+0000 7fc797a79700 1 -- 172.21.15.69:0/3848102780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7900ff400 msgr2=0x7fc790199680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.025+0000 7fc797a79700 1 --2- 172.21.15.69:0/3848102780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7900ff400 0x7fc790199680 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00d950 tx=0x7fc78c00dc60 comp rx=0 tx=0).stop 2023-07-12T15:28:22.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.025+0000 7fc796016700 1 -- 172.21.15.69:0/3848102780 reap_dead start 2023-07-12T15:28:22.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.025+0000 7fc797a79700 1 -- 172.21.15.69:0/3848102780 shutdown_connections 2023-07-12T15:28:22.029 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.025+0000 7fc797a79700 1 -- 172.21.15.69:0/3848102780 >> 172.21.15.69:0/3848102780 conn(0x7fc790076150 msgr2=0x7fc7900fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:22.030 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.025+0000 7fc797a79700 1 -- 172.21.15.69:0/3848102780 shutdown_connections 2023-07-12T15:28:22.030 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.025+0000 7fc797a79700 1 -- 172.21.15.69:0/3848102780 wait complete. 2023-07-12T15:28:22.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: Adjusting osd_memory_target on smithi069 to 3780M 2023-07-12T15:28:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:28:22.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:28:22.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: Updating smithi069:/etc/ceph/ceph.conf 2023-07-12T15:28:22.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: Updating smithi145:/etc/ceph/ceph.conf 2023-07-12T15:28:22.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: Updating smithi069:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:28:22.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: Updating smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:28:22.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2023-07-12T15:28:22.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2023-07-12T15:28:22.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2023-07-12T15:28:22.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: Adjusting osd_memory_target on smithi069 to 3780M 2023-07-12T15:28:22.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:28:22.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:28:22.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: Updating smithi069:/etc/ceph/ceph.conf 2023-07-12T15:28:22.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: Updating smithi145:/etc/ceph/ceph.conf 2023-07-12T15:28:22.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: Updating smithi069:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:28:22.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: Updating smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:28:22.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2023-07-12T15:28:22.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2023-07-12T15:28:22.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2023-07-12T15:28:22.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.782+0000 7f05b6ce2700 1 -- 172.21.15.69:0/3287505889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b00ff400 msgr2=0x7f05b00ff7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.782+0000 7f05b6ce2700 1 --2- 172.21.15.69:0/3287505889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b00ff400 0x7f05b00ff7d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009920 tx=0x7f05a4009c30 comp rx=0 tx=0).stop 2023-07-12T15:28:22.336 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.782+0000 7f05b6ce2700 1 -- 172.21.15.69:0/3287505889 shutdown_connections 2023-07-12T15:28:22.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.782+0000 7f05b6ce2700 1 --2- 172.21.15.69:0/3287505889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05b0101610 0x7f05b010c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.782+0000 7f05b6ce2700 1 --2- 172.21.15.69:0/3287505889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b00ff400 0x7f05b00ff7d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.782+0000 7f05b6ce2700 1 --2- 172.21.15.69:0/3287505889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b0100c60 0x7f05b01010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.782+0000 7f05b6ce2700 1 -- 172.21.15.69:0/3287505889 >> 172.21.15.69:0/3287505889 conn(0x7f05b0076150 msgr2=0x7f05b0076550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:22.337 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.782+0000 7f05b6ce2700 1 -- 172.21.15.69:0/3287505889 shutdown_connections 2023-07-12T15:28:22.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.782+0000 7f05b6ce2700 1 -- 172.21.15.69:0/3287505889 wait complete. 2023-07-12T15:28:22.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b6ce2700 1 Processor -- start 2023-07-12T15:28:22.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b6ce2700 1 -- start start 2023-07-12T15:28:22.338 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b6ce2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b00ff400 0x7f05b019b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b6ce2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05b0100c60 0x7f05b019bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b6ce2700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b0101610 0x7f05b0195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b6ce2700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b0196030 con 0x7f05b00ff400 2023-07-12T15:28:22.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b6ce2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b01961a0 con 0x7f05b0100c60 2023-07-12T15:28:22.339 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b6ce2700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b0196360 con 0x7f05b0101610 2023-07-12T15:28:22.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b4a7e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b00ff400 0x7f05b019b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b4a7e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b00ff400 0x7f05b019b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41270/0 (socket says 172.21.15.69:41270) 2023-07-12T15:28:22.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b4a7e700 1 -- 172.21.15.69:0/2548369208 learned_addr learned my addr 172.21.15.69:0/2548369208 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:22.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b527f700 1 --2- 172.21.15.69:0/2548369208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b0101610 0x7f05b0195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.340 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05affff700 1 --2- 172.21.15.69:0/2548369208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05b0100c60 0x7f05b019bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b4a7e700 1 -- 172.21.15.69:0/2548369208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b0101610 msgr2=0x7f05b0195880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b4a7e700 1 --2- 172.21.15.69:0/2548369208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b0101610 0x7f05b0195880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b4a7e700 1 -- 172.21.15.69:0/2548369208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05b0100c60 msgr2=0x7f05b019bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b4a7e700 1 --2- 172.21.15.69:0/2548369208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05b0100c60 0x7f05b019bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b4a7e700 1 -- 172.21.15.69:0/2548369208 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c0095f0 con 0x7f05b00ff400 2023-07-12T15:28:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.783+0000 7f05b4a7e700 1 --2- 172.21.15.69:0/2548369208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b00ff400 0x7f05b019b890 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009230 tx=0x7f05a400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:22.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.784+0000 7f05adffb700 1 -- 172.21.15.69:0/2548369208 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a401c070 con 0x7f05b00ff400 2023-07-12T15:28:22.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.784+0000 7f05adffb700 1 -- 172.21.15.69:0/2548369208 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f05a400ba70 con 0x7f05b00ff400 2023-07-12T15:28:22.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.784+0000 7f05b6ce2700 1 -- 172.21.15.69:0/2548369208 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a4009520 con 0x7f05b00ff400 2023-07-12T15:28:22.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.784+0000 7f05adffb700 1 -- 172.21.15.69:0/2548369208 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a4004930 con 0x7f05b00ff400 2023-07-12T15:28:22.342 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.784+0000 7f05b6ce2700 1 -- 172.21.15.69:0/2548369208 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b01a2640 con 0x7f05b00ff400 2023-07-12T15:28:22.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.785+0000 7f05b6ce2700 1 -- 172.21.15.69:0/2548369208 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f05b010fb60 con 0x7f05b00ff400 2023-07-12T15:28:22.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.785+0000 7f05adffb700 1 -- 172.21.15.69:0/2548369208 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f05a4020070 con 0x7f05b00ff400 2023-07-12T15:28:22.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.786+0000 7f05adffb700 1 --2- 172.21.15.69:0/2548369208 >> v2:172.21.15.69:6800/1242716914 conn(0x7f05a006c810 0x7f05a006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.786+0000 7f05adffb700 1 -- 172.21.15.69:0/2548369208 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f05a408c760 con 0x7f05b00ff400 2023-07-12T15:28:22.343 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.786+0000 7f05affff700 1 --2- 172.21.15.69:0/2548369208 >> v2:172.21.15.69:6800/1242716914 conn(0x7f05a006c810 0x7f05a006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.786+0000 7f05adffb700 1 --2- 172.21.15.69:0/2548369208 >> v2:172.21.15.145:6812/2623299546 conn(0x7f05a0075fe0 0x7f05a0078410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.786+0000 7f05adffb700 1 -- 172.21.15.69:0/2548369208 --> v2:172.21.15.145:6812/2623299546 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05a0078aa0 con 0x7f05a0075fe0 2023-07-12T15:28:22.344 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.786+0000 7f05adffb700 1 -- 172.21.15.69:0/2548369208 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05a408ee20 con 0x7f05b00ff400 2023-07-12T15:28:22.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.786+0000 7f05affff700 1 --2- 172.21.15.69:0/2548369208 >> v2:172.21.15.69:6800/1242716914 conn(0x7f05a006c810 0x7f05a006ecc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f05b01970b0 tx=0x7f059c009310 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:22.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.786+0000 7f05b527f700 1 --2- 172.21.15.69:0/2548369208 >> v2:172.21.15.145:6812/2623299546 conn(0x7f05a0075fe0 0x7f05a0078410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.787+0000 7f05b527f700 1 --2- 172.21.15.69:0/2548369208 >> v2:172.21.15.145:6812/2623299546 conn(0x7f05a0075fe0 0x7f05a0078410 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:22.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.787+0000 7f05adffb700 1 -- 172.21.15.69:0/2548369208 <== osd.7 v2:172.21.15.145:6812/2623299546 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f05a0078aa0 con 0x7f05a0075fe0 2023-07-12T15:28:22.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.803+0000 7f05b6ce2700 1 -- 172.21.15.69:0/2548369208 --> v2:172.21.15.145:6812/2623299546 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f05b004e800 con 0x7f05a0075fe0 2023-07-12T15:28:22.345 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.803+0000 7f05adffb700 1 -- 172.21.15.69:0/2548369208 <== osd.7 v2:172.21.15.145:6812/2623299546 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f05b004e800 con 0x7f05a0075fe0 2023-07-12T15:28:22.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.803+0000 7f05b6ce2700 1 -- 172.21.15.69:0/2548369208 >> v2:172.21.15.145:6812/2623299546 conn(0x7f05a0075fe0 msgr2=0x7f05a0078410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.804+0000 7f05b6ce2700 1 --2- 172.21.15.69:0/2548369208 >> v2:172.21.15.145:6812/2623299546 conn(0x7f05a0075fe0 0x7f05a0078410 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.804+0000 7f05b6ce2700 1 -- 172.21.15.69:0/2548369208 >> v2:172.21.15.69:6800/1242716914 conn(0x7f05a006c810 msgr2=0x7f05a006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.804+0000 7f05b6ce2700 1 --2- 172.21.15.69:0/2548369208 >> v2:172.21.15.69:6800/1242716914 conn(0x7f05a006c810 0x7f05a006ecc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f05b01970b0 tx=0x7f059c009310 comp rx=0 tx=0).stop 2023-07-12T15:28:22.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.804+0000 7f05b6ce2700 1 -- 172.21.15.69:0/2548369208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b00ff400 msgr2=0x7f05b019b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.804+0000 7f05b6ce2700 1 --2- 172.21.15.69:0/2548369208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b00ff400 0x7f05b019b890 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009230 tx=0x7f05a400fa00 comp rx=0 tx=0).stop 2023-07-12T15:28:22.346 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.804+0000 7f05b527f700 1 -- 172.21.15.69:0/2548369208 reap_dead start 2023-07-12T15:28:22.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.804+0000 7f05b6ce2700 1 -- 172.21.15.69:0/2548369208 shutdown_connections 2023-07-12T15:28:22.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.804+0000 7f05b6ce2700 1 -- 172.21.15.69:0/2548369208 >> 172.21.15.69:0/2548369208 conn(0x7f05b0076150 msgr2=0x7f05b00fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:22.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.804+0000 7f05b6ce2700 1 -- 172.21.15.69:0/2548369208 shutdown_connections 2023-07-12T15:28:22.347 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:20.804+0000 7f05b6ce2700 1 -- 172.21.15.69:0/2548369208 wait complete. 2023-07-12T15:28:22.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:28:22.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: Adjusting osd_memory_target on smithi069 to 3780M 2023-07-12T15:28:22.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:28:22.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:28:22.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: Updating smithi069:/etc/ceph/ceph.conf 2023-07-12T15:28:22.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: Updating smithi145:/etc/ceph/ceph.conf 2023-07-12T15:28:22.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: Updating smithi069:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:28:22.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: Updating smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-12T15:28:22.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2023-07-12T15:28:22.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2023-07-12T15:28:22.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2023-07-12T15:28:22.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.395+0000 7f8552c70700 1 -- 172.21.15.69:0/3666996135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f854c105490 msgr2=0x7f854c10d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.395+0000 7f8552c70700 1 --2- 172.21.15.69:0/3666996135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f854c105490 0x7f854c10d010 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f854400b0d0 tx=0x7f854400b3e0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.611 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.395+0000 7f8552c70700 1 -- 172.21.15.69:0/3666996135 shutdown_connections 2023-07-12T15:28:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.395+0000 7f8552c70700 1 --2- 172.21.15.69:0/3666996135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f854c105490 0x7f854c10d010 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.395+0000 7f8552c70700 1 --2- 172.21.15.69:0/3666996135 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f854c1003f0 0x7f854c1007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.395+0000 7f8552c70700 1 --2- 172.21.15.69:0/3666996135 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f854c100d90 0x7f854c104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.395+0000 7f8552c70700 1 -- 172.21.15.69:0/3666996135 >> 172.21.15.69:0/3666996135 conn(0x7f854c0fbc50 msgr2=0x7f854c0fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.395+0000 7f8552c70700 1 -- 172.21.15.69:0/3666996135 shutdown_connections 2023-07-12T15:28:22.612 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.395+0000 7f8552c70700 1 -- 172.21.15.69:0/3666996135 wait complete. 2023-07-12T15:28:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.396+0000 7f8552c70700 1 Processor -- start 2023-07-12T15:28:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.396+0000 7f8552c70700 1 -- start start 2023-07-12T15:28:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.396+0000 7f8552c70700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f854c1003f0 0x7f854c072e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.396+0000 7f8552c70700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f854c100d90 0x7f854c073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.396+0000 7f8552c70700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f854c105490 0x7f854c073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.396+0000 7f8552c70700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c1a3ee0 con 0x7f854c1003f0 2023-07-12T15:28:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.396+0000 7f8552c70700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c1a4050 con 0x7f854c100d90 2023-07-12T15:28:22.613 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.396+0000 7f8552c70700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854c1a4210 con 0x7f854c105490 2023-07-12T15:28:22.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.396+0000 7f8550a0c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f854c1003f0 0x7f854c072e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.396+0000 7f8550a0c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f854c1003f0 0x7f854c072e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41302/0 (socket says 172.21.15.69:41302) 2023-07-12T15:28:22.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.396+0000 7f8550a0c700 1 -- 172.21.15.69:0/3785923513 learned_addr learned my addr 172.21.15.69:0/3785923513 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:22.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.396+0000 7f855120d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f854c105490 0x7f854c073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.614 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.396+0000 7f854bfff700 1 --2- 172.21.15.69:0/3785923513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f854c100d90 0x7f854c073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.397+0000 7f8550a0c700 1 -- 172.21.15.69:0/3785923513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f854c105490 msgr2=0x7f854c073880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.397+0000 7f8550a0c700 1 --2- 172.21.15.69:0/3785923513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f854c105490 0x7f854c073880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.397+0000 7f8550a0c700 1 -- 172.21.15.69:0/3785923513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f854c100d90 msgr2=0x7f854c073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.397+0000 7f8550a0c700 1 --2- 172.21.15.69:0/3785923513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f854c100d90 0x7f854c073340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.397+0000 7f8550a0c700 1 -- 172.21.15.69:0/3785923513 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8540009520 con 0x7f854c1003f0 2023-07-12T15:28:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.397+0000 7f8550a0c700 1 --2- 172.21.15.69:0/3785923513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f854c1003f0 0x7f854c072e00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f854000eb30 tx=0x7f854000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.397+0000 7f8549ffb700 1 -- 172.21.15.69:0/3785923513 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540009880 con 0x7f854c1003f0 2023-07-12T15:28:22.615 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.397+0000 7f8549ffb700 1 -- 172.21.15.69:0/3785923513 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f85400044b0 con 0x7f854c1003f0 2023-07-12T15:28:22.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.397+0000 7f8552c70700 1 -- 172.21.15.69:0/3785923513 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8544009d00 con 0x7f854c1003f0 2023-07-12T15:28:22.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.397+0000 7f8549ffb700 1 -- 172.21.15.69:0/3785923513 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540003ca0 con 0x7f854c1003f0 2023-07-12T15:28:22.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.397+0000 7f8552c70700 1 -- 172.21.15.69:0/3785923513 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c1a48b0 con 0x7f854c1003f0 2023-07-12T15:28:22.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.398+0000 7f8552c70700 1 -- 172.21.15.69:0/3785923513 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f854c10fa00 con 0x7f854c1003f0 2023-07-12T15:28:22.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.398+0000 7f8549ffb700 1 -- 172.21.15.69:0/3785923513 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f8540003e00 con 0x7f854c1003f0 2023-07-12T15:28:22.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.399+0000 7f8549ffb700 1 --2- 172.21.15.69:0/3785923513 >> v2:172.21.15.69:6800/1242716914 conn(0x7f853c06c810 0x7f853c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.616 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.399+0000 7f8549ffb700 1 -- 172.21.15.69:0/3785923513 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f8540014070 con 0x7f854c1003f0 2023-07-12T15:28:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.399+0000 7f854bfff700 1 --2- 172.21.15.69:0/3785923513 >> v2:172.21.15.69:6800/1242716914 conn(0x7f853c06c810 0x7f853c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.399+0000 7f8549ffb700 1 --2- 172.21.15.69:0/3785923513 >> v2:172.21.15.69:6801/657400327 conn(0x7f853c075f90 0x7f853c0783c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.399+0000 7f8549ffb700 1 -- 172.21.15.69:0/3785923513 --> v2:172.21.15.69:6801/657400327 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f853c078a50 con 0x7f853c075f90 2023-07-12T15:28:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.399+0000 7f8549ffb700 1 -- 172.21.15.69:0/3785923513 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7f854008e520 con 0x7f854c1003f0 2023-07-12T15:28:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.399+0000 7f855120d700 1 --2- 172.21.15.69:0/3785923513 >> v2:172.21.15.69:6801/657400327 conn(0x7f853c075f90 0x7f853c0783c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.399+0000 7f854bfff700 1 --2- 172.21.15.69:0/3785923513 >> v2:172.21.15.69:6800/1242716914 conn(0x7f853c06c810 0x7f853c06ecc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f854c0745f0 tx=0x7f85380073d0 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:22.617 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.399+0000 7f855120d700 1 --2- 172.21.15.69:0/3785923513 >> v2:172.21.15.69:6801/657400327 conn(0x7f853c075f90 0x7f853c0783c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.400+0000 7f8549ffb700 1 -- 172.21.15.69:0/3785923513 <== osd.0 v2:172.21.15.69:6801/657400327 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f853c078a50 con 0x7f853c075f90 2023-07-12T15:28:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.417+0000 7f8552c70700 1 -- 172.21.15.69:0/3785923513 --> v2:172.21.15.69:6801/657400327 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f854c04e800 con 0x7f853c075f90 2023-07-12T15:28:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.418+0000 7f8549ffb700 1 -- 172.21.15.69:0/3785923513 <== osd.0 v2:172.21.15.69:6801/657400327 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f854c04e800 con 0x7f853c075f90 2023-07-12T15:28:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.418+0000 7f8552c70700 1 -- 172.21.15.69:0/3785923513 >> v2:172.21.15.69:6801/657400327 conn(0x7f853c075f90 msgr2=0x7f853c0783c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.418+0000 7f8552c70700 1 --2- 172.21.15.69:0/3785923513 >> v2:172.21.15.69:6801/657400327 conn(0x7f853c075f90 0x7f853c0783c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.418+0000 7f8552c70700 1 -- 172.21.15.69:0/3785923513 >> v2:172.21.15.69:6800/1242716914 conn(0x7f853c06c810 msgr2=0x7f853c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.619 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.418+0000 7f8552c70700 1 --2- 172.21.15.69:0/3785923513 >> v2:172.21.15.69:6800/1242716914 conn(0x7f853c06c810 0x7f853c06ecc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f854c0745f0 tx=0x7f85380073d0 comp rx=0 tx=0).stop 2023-07-12T15:28:22.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.418+0000 7f8552c70700 1 -- 172.21.15.69:0/3785923513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f854c1003f0 msgr2=0x7f854c072e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:22.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.418+0000 7f8552c70700 1 --2- 172.21.15.69:0/3785923513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f854c1003f0 0x7f854c072e00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f854000eb30 tx=0x7f854000ee40 comp rx=0 tx=0).stop 2023-07-12T15:28:22.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.418+0000 7f855120d700 1 -- 172.21.15.69:0/3785923513 reap_dead start 2023-07-12T15:28:22.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.418+0000 7f8552c70700 1 -- 172.21.15.69:0/3785923513 shutdown_connections 2023-07-12T15:28:22.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.418+0000 7f8552c70700 1 -- 172.21.15.69:0/3785923513 >> 172.21.15.69:0/3785923513 conn(0x7f854c0fbc50 msgr2=0x7f854c0fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:22.620 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.418+0000 7f8552c70700 1 -- 172.21.15.69:0/3785923513 shutdown_connections 2023-07-12T15:28:22.621 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:22.418+0000 7f8552c70700 1 -- 172.21.15.69:0/3785923513 wait complete. 2023-07-12T15:28:22.898 INFO:teuthology.orchestra.run.smithi069.stdout:55834574887 2023-07-12T15:28:22.898 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.1 2023-07-12T15:28:23.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:23 smithi069 ceph-mon[102777]: Updating smithi069:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.client.admin.keyring 2023-07-12T15:28:23.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:23 smithi069 ceph-mon[102777]: Updating smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.client.admin.keyring 2023-07-12T15:28:23.133 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:23 smithi069 ceph-mon[102777]: Deploying daemon alertmanager.a on smithi069 2023-07-12T15:28:23.133 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:23 smithi069 ceph-mon[94241]: Updating smithi069:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.client.admin.keyring 2023-07-12T15:28:23.134 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:23 smithi069 ceph-mon[94241]: Updating smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.client.admin.keyring 2023-07-12T15:28:23.134 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:23 smithi069 ceph-mon[94241]: Deploying daemon alertmanager.a on smithi069 2023-07-12T15:28:23.195 INFO:teuthology.orchestra.run.smithi069.stdout:77309411362 2023-07-12T15:28:23.196 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.2 2023-07-12T15:28:23.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:23 smithi145 ceph-mon[95801]: Updating smithi069:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.client.admin.keyring 2023-07-12T15:28:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:23 smithi145 ceph-mon[95801]: Updating smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/config/ceph.client.admin.keyring 2023-07-12T15:28:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:23 smithi145 ceph-mon[95801]: Deploying daemon alertmanager.a on smithi069 2023-07-12T15:28:23.352 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.328+0000 7f60ed07a700 1 -- 172.21.15.69:0/369598210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60e810a860 msgr2=0x7f60e810ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:23.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.328+0000 7f60ed07a700 1 --2- 172.21.15.69:0/369598210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60e810a860 0x7f60e810ccb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f60e000b0a0 tx=0x7f60e000b3b0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.328+0000 7f60ed07a700 1 -- 172.21.15.69:0/369598210 shutdown_connections 2023-07-12T15:28:23.353 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.328+0000 7f60ed07a700 1 --2- 172.21.15.69:0/369598210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60e810a860 0x7f60e810ccb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.328+0000 7f60ed07a700 1 --2- 172.21.15.69:0/369598210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60e81013e0 0x7f60e81017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.328+0000 7f60ed07a700 1 --2- 172.21.15.69:0/369598210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60e8101d80 0x7f60e810a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.328+0000 7f60ed07a700 1 -- 172.21.15.69:0/369598210 >> 172.21.15.69:0/369598210 conn(0x7f60e8076150 msgr2=0x7f60e8076550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:23.354 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.328+0000 7f60ed07a700 1 -- 172.21.15.69:0/369598210 shutdown_connections 2023-07-12T15:28:23.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.328+0000 7f60ed07a700 1 -- 172.21.15.69:0/369598210 wait complete. 2023-07-12T15:28:23.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.329+0000 7f60ed07a700 1 Processor -- start 2023-07-12T15:28:23.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.329+0000 7f60ed07a700 1 -- start start 2023-07-12T15:28:23.355 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60ed07a700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60e81013e0 0x7f60e8199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:23.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60ed07a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60e8101d80 0x7f60e8199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:23.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60ed07a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60e810a860 0x7f60e8193980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:23.356 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60ed07a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e8193ff0 con 0x7f60e810a860 2023-07-12T15:28:23.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60ed07a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e8194160 con 0x7f60e8101d80 2023-07-12T15:28:23.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60ed07a700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e8194320 con 0x7f60e81013e0 2023-07-12T15:28:23.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60e759e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60e810a860 0x7f60e8193980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:23.357 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60e759e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60e810a860 0x7f60e8193980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41322/0 (socket says 172.21.15.69:41322) 2023-07-12T15:28:23.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60e759e700 1 -- 172.21.15.69:0/3852912575 learned_addr learned my addr 172.21.15.69:0/3852912575 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:23.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60e6d9d700 1 --2- 172.21.15.69:0/3852912575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60e81013e0 0x7f60e8199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:23.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60e659c700 1 --2- 172.21.15.69:0/3852912575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60e8101d80 0x7f60e8199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:23.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60e759e700 1 -- 172.21.15.69:0/3852912575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60e81013e0 msgr2=0x7f60e8199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:23.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60e759e700 1 --2- 172.21.15.69:0/3852912575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60e81013e0 0x7f60e8199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.358 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60e759e700 1 -- 172.21.15.69:0/3852912575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60e8101d80 msgr2=0x7f60e8199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:23.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60e759e700 1 --2- 172.21.15.69:0/3852912575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60e8101d80 0x7f60e8199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60e759e700 1 -- 172.21.15.69:0/3852912575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e0009d00 con 0x7f60e810a860 2023-07-12T15:28:23.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.330+0000 7f60e759e700 1 --2- 172.21.15.69:0/3852912575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60e810a860 0x7f60e8193980 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f60e000b780 tx=0x7f60e000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:23.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.331+0000 7f60cffff700 1 -- 172.21.15.69:0/3852912575 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e000e040 con 0x7f60e810a860 2023-07-12T15:28:23.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.331+0000 7f60cffff700 1 -- 172.21.15.69:0/3852912575 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f60e0004e60 con 0x7f60e810a860 2023-07-12T15:28:23.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.331+0000 7f60cffff700 1 -- 172.21.15.69:0/3852912575 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e0005040 con 0x7f60e810a860 2023-07-12T15:28:23.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.331+0000 7f60ed07a700 1 -- 172.21.15.69:0/3852912575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e81945a0 con 0x7f60e810a860 2023-07-12T15:28:23.359 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.331+0000 7f60ed07a700 1 -- 172.21.15.69:0/3852912575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e81a08f0 con 0x7f60e810a860 2023-07-12T15:28:23.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.332+0000 7f60ed07a700 1 -- 172.21.15.69:0/3852912575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f60e8100f70 con 0x7f60e810a860 2023-07-12T15:28:23.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.333+0000 7f60cffff700 1 -- 172.21.15.69:0/3852912575 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f60e001d470 con 0x7f60e810a860 2023-07-12T15:28:23.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.333+0000 7f60cffff700 1 --2- 172.21.15.69:0/3852912575 >> v2:172.21.15.69:6800/1242716914 conn(0x7f60d006c810 0x7f60d006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:23.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.333+0000 7f60cffff700 1 -- 172.21.15.69:0/3852912575 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f60e008d050 con 0x7f60e810a860 2023-07-12T15:28:23.360 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.333+0000 7f60e6d9d700 1 --2- 172.21.15.69:0/3852912575 >> v2:172.21.15.69:6800/1242716914 conn(0x7f60d006c810 0x7f60d006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.333+0000 7f60cffff700 1 --2- 172.21.15.69:0/3852912575 >> v2:172.21.15.145:6800/184109944 conn(0x7f60d0075fe0 0x7f60d0078410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.333+0000 7f60cffff700 1 -- 172.21.15.69:0/3852912575 --> v2:172.21.15.145:6800/184109944 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60d0078aa0 con 0x7f60d0075fe0 2023-07-12T15:28:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.333+0000 7f60cffff700 1 -- 172.21.15.69:0/3852912575 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60e008f710 con 0x7f60e810a860 2023-07-12T15:28:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.334+0000 7f60e6d9d700 1 --2- 172.21.15.69:0/3852912575 >> v2:172.21.15.69:6800/1242716914 conn(0x7f60d006c810 0x7f60d006ecc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60d8007be0 tx=0x7f60d80073d0 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.334+0000 7f60e659c700 1 --2- 172.21.15.69:0/3852912575 >> v2:172.21.15.145:6800/184109944 conn(0x7f60d0075fe0 0x7f60d0078410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.334+0000 7f60e659c700 1 --2- 172.21.15.69:0/3852912575 >> v2:172.21.15.145:6800/184109944 conn(0x7f60d0075fe0 0x7f60d0078410 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:23.361 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.335+0000 7f60cffff700 1 -- 172.21.15.69:0/3852912575 <== osd.4 v2:172.21.15.145:6800/184109944 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f60d0078aa0 con 0x7f60d0075fe0 2023-07-12T15:28:23.362 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.351+0000 7f60ed07a700 1 -- 172.21.15.69:0/3852912575 --> v2:172.21.15.145:6800/184109944 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f60e8061180 con 0x7f60d0075fe0 2023-07-12T15:28:23.363 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.352+0000 7f60cffff700 1 -- 172.21.15.69:0/3852912575 <== osd.4 v2:172.21.15.145:6800/184109944 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f60e8061180 con 0x7f60d0075fe0 2023-07-12T15:28:23.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.352+0000 7f60ed07a700 1 -- 172.21.15.69:0/3852912575 >> v2:172.21.15.145:6800/184109944 conn(0x7f60d0075fe0 msgr2=0x7f60d0078410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:23.364 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.352+0000 7f60ed07a700 1 --2- 172.21.15.69:0/3852912575 >> v2:172.21.15.145:6800/184109944 conn(0x7f60d0075fe0 0x7f60d0078410 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.352+0000 7f60ed07a700 1 -- 172.21.15.69:0/3852912575 >> v2:172.21.15.69:6800/1242716914 conn(0x7f60d006c810 msgr2=0x7f60d006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:23.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.352+0000 7f60ed07a700 1 --2- 172.21.15.69:0/3852912575 >> v2:172.21.15.69:6800/1242716914 conn(0x7f60d006c810 0x7f60d006ecc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60d8007be0 tx=0x7f60d80073d0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.352+0000 7f60ed07a700 1 -- 172.21.15.69:0/3852912575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60e810a860 msgr2=0x7f60e8193980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:23.365 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.352+0000 7f60ed07a700 1 --2- 172.21.15.69:0/3852912575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60e810a860 0x7f60e8193980 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f60e000b780 tx=0x7f60e000bd60 comp rx=0 tx=0).stop 2023-07-12T15:28:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.352+0000 7f60e759e700 1 -- 172.21.15.69:0/3852912575 reap_dead start 2023-07-12T15:28:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.352+0000 7f60ed07a700 1 -- 172.21.15.69:0/3852912575 shutdown_connections 2023-07-12T15:28:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.352+0000 7f60ed07a700 1 -- 172.21.15.69:0/3852912575 >> 172.21.15.69:0/3852912575 conn(0x7f60e8076150 msgr2=0x7f60e810b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.352+0000 7f60ed07a700 1 -- 172.21.15.69:0/3852912575 shutdown_connections 2023-07-12T15:28:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.352+0000 7f60ed07a700 1 -- 172.21.15.69:0/3852912575 wait complete. 2023-07-12T15:28:23.585 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.375+0000 7fe695ce6700 1 -- 172.21.15.69:0/2284522683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe69010a730 msgr2=0x7fe69010cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:23.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.375+0000 7fe695ce6700 1 --2- 172.21.15.69:0/2284522683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe69010a730 0x7fe69010cb70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe688009f90 tx=0x7fe68800b330 comp rx=0 tx=0).stop 2023-07-12T15:28:23.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.375+0000 7fe695ce6700 1 -- 172.21.15.69:0/2284522683 shutdown_connections 2023-07-12T15:28:23.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.375+0000 7fe695ce6700 1 --2- 172.21.15.69:0/2284522683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe69010a730 0x7fe69010cb70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.375+0000 7fe695ce6700 1 --2- 172.21.15.69:0/2284522683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6901013e0 0x7fe6901017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.375+0000 7fe695ce6700 1 --2- 172.21.15.69:0/2284522683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe690101cf0 0x7fe69010a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.375+0000 7fe695ce6700 1 -- 172.21.15.69:0/2284522683 >> 172.21.15.69:0/2284522683 conn(0x7fe690076160 msgr2=0x7fe690076560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:23.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.375+0000 7fe695ce6700 1 -- 172.21.15.69:0/2284522683 shutdown_connections 2023-07-12T15:28:23.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.375+0000 7fe695ce6700 1 -- 172.21.15.69:0/2284522683 wait complete. 2023-07-12T15:28:23.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.376+0000 7fe695ce6700 1 Processor -- start 2023-07-12T15:28:23.587 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.376+0000 7fe695ce6700 1 -- start start 2023-07-12T15:28:23.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe695ce6700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6901013e0 0x7fe69019b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:23.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe695ce6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe690101cf0 0x7fe69019bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:23.588 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe695ce6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69010a730 0x7fe6901958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:23.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe695ce6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe690195f50 con 0x7fe69010a730 2023-07-12T15:28:23.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe695ce6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6901960c0 con 0x7fe690101cf0 2023-07-12T15:28:23.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe695ce6700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe690196280 con 0x7fe6901013e0 2023-07-12T15:28:23.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe68ffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69010a730 0x7fe6901958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:23.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe68ffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69010a730 0x7fe6901958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41338/0 (socket says 172.21.15.69:41338) 2023-07-12T15:28:23.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe68ffff700 1 -- 172.21.15.69:0/1467291384 learned_addr learned my addr 172.21.15.69:0/1467291384 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:23.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe68f7fe700 1 --2- 172.21.15.69:0/1467291384 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6901013e0 0x7fe69019b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:23.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe68effd700 1 --2- 172.21.15.69:0/1467291384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe690101cf0 0x7fe69019bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:23.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe68ffff700 1 -- 172.21.15.69:0/1467291384 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6901013e0 msgr2=0x7fe69019b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:23.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe68ffff700 1 --2- 172.21.15.69:0/1467291384 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6901013e0 0x7fe69019b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe68ffff700 1 -- 172.21.15.69:0/1467291384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe690101cf0 msgr2=0x7fe69019bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:23.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe68ffff700 1 --2- 172.21.15.69:0/1467291384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe690101cf0 0x7fe69019bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe68ffff700 1 -- 172.21.15.69:0/1467291384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6800095f0 con 0x7fe69010a730 2023-07-12T15:28:23.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe68f7fe700 1 --2- 172.21.15.69:0/1467291384 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6901013e0 0x7fe69019b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:28:23.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe68ffff700 1 --2- 172.21.15.69:0/1467291384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69010a730 0x7fe6901958e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe68800b6c0 tx=0x7fe68800bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:23.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.377+0000 7fe68cff9700 1 -- 172.21.15.69:0/1467291384 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68800e040 con 0x7fe69010a730 2023-07-12T15:28:23.593 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.378+0000 7fe68cff9700 1 -- 172.21.15.69:0/1467291384 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fe688004e90 con 0x7fe69010a730 2023-07-12T15:28:23.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.378+0000 7fe68cff9700 1 -- 172.21.15.69:0/1467291384 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68801b450 con 0x7fe69010a730 2023-07-12T15:28:23.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.378+0000 7fe695ce6700 1 -- 172.21.15.69:0/1467291384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe688009c40 con 0x7fe69010a730 2023-07-12T15:28:23.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.378+0000 7fe695ce6700 1 -- 172.21.15.69:0/1467291384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6901a2680 con 0x7fe69010a730 2023-07-12T15:28:23.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.378+0000 7fe695ce6700 1 -- 172.21.15.69:0/1467291384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe69010f9a0 con 0x7fe69010a730 2023-07-12T15:28:23.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.379+0000 7fe68cff9700 1 -- 172.21.15.69:0/1467291384 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7fe688007c40 con 0x7fe69010a730 2023-07-12T15:28:23.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.379+0000 7fe68cff9700 1 --2- 172.21.15.69:0/1467291384 >> v2:172.21.15.69:6800/1242716914 conn(0x7fe67806c810 0x7fe67806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:23.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.380+0000 7fe68cff9700 1 -- 172.21.15.69:0/1467291384 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fe68808d8d0 con 0x7fe69010a730 2023-07-12T15:28:23.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.380+0000 7fe68f7fe700 1 --2- 172.21.15.69:0/1467291384 >> v2:172.21.15.69:6800/1242716914 conn(0x7fe67806c810 0x7fe67806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:23.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.380+0000 7fe68cff9700 1 --2- 172.21.15.69:0/1467291384 >> v2:172.21.15.69:6813/2610582937 conn(0x7fe678076060 0x7fe678078490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:23.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.380+0000 7fe68cff9700 1 -- 172.21.15.69:0/1467291384 --> v2:172.21.15.69:6813/2610582937 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe678078b20 con 0x7fe678076060 2023-07-12T15:28:23.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.380+0000 7fe68cff9700 1 -- 172.21.15.69:0/1467291384 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe68808dd70 con 0x7fe69010a730 2023-07-12T15:28:23.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.380+0000 7fe68effd700 1 --2- 172.21.15.69:0/1467291384 >> v2:172.21.15.69:6813/2610582937 conn(0x7fe678076060 0x7fe678078490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:23.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.380+0000 7fe68f7fe700 1 --2- 172.21.15.69:0/1467291384 >> v2:172.21.15.69:6800/1242716914 conn(0x7fe67806c810 0x7fe67806ecc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe680005fd0 tx=0x7fe680009310 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:23.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.380+0000 7fe68effd700 1 --2- 172.21.15.69:0/1467291384 >> v2:172.21.15.69:6813/2610582937 conn(0x7fe678076060 0x7fe678078490 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:23.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.381+0000 7fe68cff9700 1 -- 172.21.15.69:0/1467291384 <== osd.3 v2:172.21.15.69:6813/2610582937 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe678078b20 con 0x7fe678076060 2023-07-12T15:28:23.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.399+0000 7fe695ce6700 1 -- 172.21.15.69:0/1467291384 --> v2:172.21.15.69:6813/2610582937 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe690061180 con 0x7fe678076060 2023-07-12T15:28:23.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.399+0000 7fe68cff9700 1 -- 172.21.15.69:0/1467291384 <== osd.3 v2:172.21.15.69:6813/2610582937 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe690061180 con 0x7fe678076060 2023-07-12T15:28:23.598 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.399+0000 7fe695ce6700 1 -- 172.21.15.69:0/1467291384 >> v2:172.21.15.69:6813/2610582937 conn(0x7fe678076060 msgr2=0x7fe678078490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.399+0000 7fe695ce6700 1 --2- 172.21.15.69:0/1467291384 >> v2:172.21.15.69:6813/2610582937 conn(0x7fe678076060 0x7fe678078490 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.400+0000 7fe695ce6700 1 -- 172.21.15.69:0/1467291384 >> v2:172.21.15.69:6800/1242716914 conn(0x7fe67806c810 msgr2=0x7fe67806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.400+0000 7fe695ce6700 1 --2- 172.21.15.69:0/1467291384 >> v2:172.21.15.69:6800/1242716914 conn(0x7fe67806c810 0x7fe67806ecc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe680005fd0 tx=0x7fe680009310 comp rx=0 tx=0).stop 2023-07-12T15:28:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.400+0000 7fe695ce6700 1 -- 172.21.15.69:0/1467291384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69010a730 msgr2=0x7fe6901958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.400+0000 7fe695ce6700 1 --2- 172.21.15.69:0/1467291384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69010a730 0x7fe6901958e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe68800b6c0 tx=0x7fe68800bee0 comp rx=0 tx=0).stop 2023-07-12T15:28:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.400+0000 7fe68ffff700 1 -- 172.21.15.69:0/1467291384 reap_dead start 2023-07-12T15:28:23.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.400+0000 7fe695ce6700 1 -- 172.21.15.69:0/1467291384 shutdown_connections 2023-07-12T15:28:23.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.400+0000 7fe695ce6700 1 -- 172.21.15.69:0/1467291384 >> 172.21.15.69:0/1467291384 conn(0x7fe690076160 msgr2=0x7fe6901020c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:23.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.400+0000 7fe695ce6700 1 -- 172.21.15.69:0/1467291384 shutdown_connections 2023-07-12T15:28:23.600 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:23.400+0000 7fe695ce6700 1 -- 172.21.15.69:0/1467291384 wait complete. 2023-07-12T15:28:23.801 INFO:teuthology.orchestra.run.smithi069.stdout:231928233996 2023-07-12T15:28:23.802 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.7 2023-07-12T15:28:24.226 INFO:teuthology.orchestra.run.smithi069.stdout:34359738411 2023-07-12T15:28:24.226 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.0 2023-07-12T15:28:24.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:24 smithi145 ceph-mon[95801]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2023-07-12T15:28:24.363 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:24 smithi069 ceph-mon[102777]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2023-07-12T15:28:24.364 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:24 smithi069 ceph-mon[94241]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2023-07-12T15:28:24.364 INFO:teuthology.orchestra.run.smithi069.stdout:128849018906 2023-07-12T15:28:24.365 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.4 2023-07-12T15:28:24.766 INFO:teuthology.orchestra.run.smithi069.stdout:107374182430 2023-07-12T15:28:24.766 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.3 2023-07-12T15:28:24.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.783+0000 7f9d6a981700 1 -- 172.21.15.69:0/2159532513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d6410c760 msgr2=0x7f9d6410cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:24.786 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.783+0000 7f9d6a981700 1 --2- 172.21.15.69:0/2159532513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d6410c760 0x7f9d6410cb30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009850 tx=0x7f9d54009b60 comp rx=0 tx=0).stop 2023-07-12T15:28:24.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.784+0000 7f9d6a981700 1 -- 172.21.15.69:0/2159532513 shutdown_connections 2023-07-12T15:28:24.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.784+0000 7f9d6a981700 1 --2- 172.21.15.69:0/2159532513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d6406d7d0 0x7f9d64078d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:24.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.784+0000 7f9d6a981700 1 --2- 172.21.15.69:0/2159532513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d6410c760 0x7f9d6410cb30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:24.787 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.784+0000 7f9d6a981700 1 --2- 172.21.15.69:0/2159532513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d6410d100 0x7f9d6406d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:24.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.784+0000 7f9d6a981700 1 -- 172.21.15.69:0/2159532513 >> 172.21.15.69:0/2159532513 conn(0x7f9d6406c430 msgr2=0x7f9d64071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:24.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.784+0000 7f9d6a981700 1 -- 172.21.15.69:0/2159532513 shutdown_connections 2023-07-12T15:28:24.788 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.784+0000 7f9d6a981700 1 -- 172.21.15.69:0/2159532513 wait complete. 2023-07-12T15:28:24.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.785+0000 7f9d6a981700 1 Processor -- start 2023-07-12T15:28:24.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.785+0000 7f9d6a981700 1 -- start start 2023-07-12T15:28:24.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.785+0000 7f9d6a981700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d6406d7d0 0x7f9d64071d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:24.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.785+0000 7f9d6a981700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d6410c760 0x7f9d640722d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:24.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.785+0000 7f9d6a981700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d64072810 0x7f9d64076910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:24.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.785+0000 7f9d6a981700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d640790b0 con 0x7f9d6410c760 2023-07-12T15:28:24.789 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.785+0000 7f9d6a981700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d64076e50 con 0x7f9d64072810 2023-07-12T15:28:24.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.785+0000 7f9d6a981700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d64077010 con 0x7f9d6406d7d0 2023-07-12T15:28:24.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.785+0000 7f9d637fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d6410c760 0x7f9d640722d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:24.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.785+0000 7f9d637fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d6410c760 0x7f9d640722d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41392/0 (socket says 172.21.15.69:41392) 2023-07-12T15:28:24.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.785+0000 7f9d637fe700 1 -- 172.21.15.69:0/3693589324 learned_addr learned my addr 172.21.15.69:0/3693589324 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:24.790 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.786+0000 7f9d68f1e700 1 --2- 172.21.15.69:0/3693589324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d64072810 0x7f9d64076910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:24.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.786+0000 7f9d637fe700 1 -- 172.21.15.69:0/3693589324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d6406d7d0 msgr2=0x7f9d64071d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:24.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.786+0000 7f9d637fe700 1 --2- 172.21.15.69:0/3693589324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d6406d7d0 0x7f9d64071d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:24.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.786+0000 7f9d637fe700 1 -- 172.21.15.69:0/3693589324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d64072810 msgr2=0x7f9d64076910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:24.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.786+0000 7f9d637fe700 1 --2- 172.21.15.69:0/3693589324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d64072810 0x7f9d64076910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:24.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.786+0000 7f9d637fe700 1 -- 172.21.15.69:0/3693589324 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d540094c0 con 0x7f9d6410c760 2023-07-12T15:28:24.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.786+0000 7f9d637fe700 1 --2- 172.21.15.69:0/3693589324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d6410c760 0x7f9d640722d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800a960 tx=0x7f9d5800ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:24.791 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.787+0000 7f9d617fa700 1 -- 172.21.15.69:0/3693589324 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d58010070 con 0x7f9d6410c760 2023-07-12T15:28:24.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.787+0000 7f9d617fa700 1 -- 172.21.15.69:0/3693589324 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f9d58007d50 con 0x7f9d6410c760 2023-07-12T15:28:24.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.787+0000 7f9d617fa700 1 -- 172.21.15.69:0/3693589324 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d58014d90 con 0x7f9d6410c760 2023-07-12T15:28:24.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.788+0000 7f9d6a981700 1 -- 172.21.15.69:0/3693589324 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d640772f0 con 0x7f9d6410c760 2023-07-12T15:28:24.792 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.788+0000 7f9d6a981700 1 -- 172.21.15.69:0/3693589324 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d64077770 con 0x7f9d6410c760 2023-07-12T15:28:24.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.789+0000 7f9d6a981700 1 -- 172.21.15.69:0/3693589324 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9d6404e800 con 0x7f9d6410c760 2023-07-12T15:28:24.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.790+0000 7f9d617fa700 1 -- 172.21.15.69:0/3693589324 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f9d580078c0 con 0x7f9d6410c760 2023-07-12T15:28:24.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.791+0000 7f9d617fa700 1 --2- 172.21.15.69:0/3693589324 >> v2:172.21.15.69:6800/1242716914 conn(0x7f9d4c06c810 0x7f9d4c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:24.793 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.791+0000 7f9d617fa700 1 -- 172.21.15.69:0/3693589324 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f9d5808b040 con 0x7f9d6410c760 2023-07-12T15:28:24.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.791+0000 7f9d63fff700 1 --2- 172.21.15.69:0/3693589324 >> v2:172.21.15.69:6800/1242716914 conn(0x7f9d4c06c810 0x7f9d4c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:24.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.791+0000 7f9d617fa700 1 --2- 172.21.15.69:0/3693589324 >> v2:172.21.15.145:6808/3456495522 conn(0x7f9d4c076060 0x7f9d4c078490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:24.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.791+0000 7f9d617fa700 1 -- 172.21.15.69:0/3693589324 --> v2:172.21.15.145:6808/3456495522 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d4c078b20 con 0x7f9d4c076060 2023-07-12T15:28:24.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.791+0000 7f9d617fa700 1 -- 172.21.15.69:0/3693589324 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d5808d700 con 0x7f9d6410c760 2023-07-12T15:28:24.794 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.792+0000 7f9d63fff700 1 --2- 172.21.15.69:0/3693589324 >> v2:172.21.15.69:6800/1242716914 conn(0x7f9d4c06c810 0x7f9d4c06ecc0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9d54006010 tx=0x7f9d5400b7d0 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:24.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.792+0000 7f9d68f1e700 1 --2- 172.21.15.69:0/3693589324 >> v2:172.21.15.145:6808/3456495522 conn(0x7f9d4c076060 0x7f9d4c078490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:24.795 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.793+0000 7f9d68f1e700 1 --2- 172.21.15.69:0/3693589324 >> v2:172.21.15.145:6808/3456495522 conn(0x7f9d4c076060 0x7f9d4c078490 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:24.796 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.794+0000 7f9d617fa700 1 -- 172.21.15.69:0/3693589324 <== osd.6 v2:172.21.15.145:6808/3456495522 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9d4c078b20 con 0x7f9d4c076060 2023-07-12T15:28:24.824 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.821+0000 7f9d6a981700 1 -- 172.21.15.69:0/3693589324 --> v2:172.21.15.145:6808/3456495522 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9d6407b560 con 0x7f9d4c076060 2023-07-12T15:28:24.824 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.822+0000 7f9d617fa700 1 -- 172.21.15.69:0/3693589324 <== osd.6 v2:172.21.15.145:6808/3456495522 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9d6407b560 con 0x7f9d4c076060 2023-07-12T15:28:24.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.824+0000 7f9d6a981700 1 -- 172.21.15.69:0/3693589324 >> v2:172.21.15.145:6808/3456495522 conn(0x7f9d4c076060 msgr2=0x7f9d4c078490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:24.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.824+0000 7f9d6a981700 1 --2- 172.21.15.69:0/3693589324 >> v2:172.21.15.145:6808/3456495522 conn(0x7f9d4c076060 0x7f9d4c078490 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:24.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.824+0000 7f9d6a981700 1 -- 172.21.15.69:0/3693589324 >> v2:172.21.15.69:6800/1242716914 conn(0x7f9d4c06c810 msgr2=0x7f9d4c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:24.828 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.824+0000 7f9d6a981700 1 --2- 172.21.15.69:0/3693589324 >> v2:172.21.15.69:6800/1242716914 conn(0x7f9d4c06c810 0x7f9d4c06ecc0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9d54006010 tx=0x7f9d5400b7d0 comp rx=0 tx=0).stop 2023-07-12T15:28:24.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.824+0000 7f9d6a981700 1 -- 172.21.15.69:0/3693589324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d6410c760 msgr2=0x7f9d640722d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:24.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.824+0000 7f9d6a981700 1 --2- 172.21.15.69:0/3693589324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d6410c760 0x7f9d640722d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800a960 tx=0x7f9d5800ad20 comp rx=0 tx=0).stop 2023-07-12T15:28:24.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.824+0000 7f9d68f1e700 1 -- 172.21.15.69:0/3693589324 reap_dead start 2023-07-12T15:28:24.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.825+0000 7f9d6a981700 1 -- 172.21.15.69:0/3693589324 shutdown_connections 2023-07-12T15:28:24.829 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.825+0000 7f9d6a981700 1 -- 172.21.15.69:0/3693589324 >> 172.21.15.69:0/3693589324 conn(0x7f9d6406c430 msgr2=0x7f9d6410ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.828+0000 7f9d6a981700 1 -- 172.21.15.69:0/3693589324 shutdown_connections 2023-07-12T15:28:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:24.828+0000 7f9d6a981700 1 -- 172.21.15.69:0/3693589324 wait complete. 2023-07-12T15:28:24.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:28:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:28:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:24.879 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:25.009 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:25.010 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:25 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:25.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:25 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:25.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:25 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:25.442 INFO:teuthology.orchestra.run.smithi069.stdout:197568495634 2023-07-12T15:28:25.443 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph osd last-stat-seq osd.6 2023-07-12T15:28:25.702 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:25.702 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:25.704 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:26.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:26 smithi145 ceph-mon[95801]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2023-07-12T15:28:26.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:26 smithi069 ceph-mon[102777]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2023-07-12T15:28:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:26 smithi069 ceph-mon[94241]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2023-07-12T15:28:26.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:28:26] "GET /metrics HTTP/1.1" 200 35137 "" "Prometheus/2.43.0" 2023-07-12T15:28:28.126 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:28.126 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:28.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:28 smithi069 ceph-mon[102777]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:28.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:28 smithi069 ceph-mon[94241]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:28.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:28 smithi145 ceph-mon[95801]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:29.431 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:28:29.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:28:29.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:28:30.597 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:30 smithi069 ceph-mon[94241]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:30.598 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:30 smithi069 ceph-mon[102777]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:30.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.266+0000 7f75b2b09700 1 -- 172.21.15.69:0/1025571095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75ac105130 msgr2=0x7f75ac105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:30.718 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.266+0000 7f75b2b09700 1 --2- 172.21.15.69:0/1025571095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75ac105130 0x7f75ac105500 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009990 tx=0x7f75a0009ca0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.266+0000 7f75b2b09700 1 -- 172.21.15.69:0/1025571095 shutdown_connections 2023-07-12T15:28:30.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.266+0000 7f75b2b09700 1 --2- 172.21.15.69:0/1025571095 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75ac0ff080 0x7f75ac10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.266+0000 7f75b2b09700 1 --2- 172.21.15.69:0/1025571095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75ac105130 0x7f75ac105500 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.266+0000 7f75b2b09700 1 --2- 172.21.15.69:0/1025571095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75ac0fe6d0 0x7f75ac0feb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.266+0000 7f75b2b09700 1 -- 172.21.15.69:0/1025571095 >> 172.21.15.69:0/1025571095 conn(0x7f75ac075740 msgr2=0x7f75ac075b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:30.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.266+0000 7f75b2b09700 1 -- 172.21.15.69:0/1025571095 shutdown_connections 2023-07-12T15:28:30.719 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.267+0000 7f75b2b09700 1 -- 172.21.15.69:0/1025571095 wait complete. 2023-07-12T15:28:30.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.267+0000 7f75b2b09700 1 Processor -- start 2023-07-12T15:28:30.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.267+0000 7f75b2b09700 1 -- start start 2023-07-12T15:28:30.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.267+0000 7f75b2b09700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75ac0fe6d0 0x7f75ac19b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:30.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.267+0000 7f75b2b09700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75ac0ff080 0x7f75ac19bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:30.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.267+0000 7f75b2b09700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75ac105130 0x7f75ac195700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:30.720 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.267+0000 7f75b2b09700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac195db0 con 0x7f75ac0fe6d0 2023-07-12T15:28:30.721 INFO:teuthology.orchestra.run.smithi069.stdout:55834574889 2023-07-12T15:28:30.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.267+0000 7f75b2b09700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac195f20 con 0x7f75ac105130 2023-07-12T15:28:30.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.267+0000 7f75b2b09700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac1960e0 con 0x7f75ac0ff080 2023-07-12T15:28:30.721 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.267+0000 7f75b08a5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75ac0fe6d0 0x7f75ac19b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:30.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.267+0000 7f75abfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75ac0ff080 0x7f75ac19bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:30.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.267+0000 7f75b08a5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75ac0fe6d0 0x7f75ac19b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41408/0 (socket says 172.21.15.69:41408) 2023-07-12T15:28:30.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.267+0000 7f75b08a5700 1 -- 172.21.15.69:0/199530266 learned_addr learned my addr 172.21.15.69:0/199530266 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:30.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.268+0000 7f75b08a5700 1 -- 172.21.15.69:0/199530266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75ac0ff080 msgr2=0x7f75ac19bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:30.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.268+0000 7f75b08a5700 1 --2- 172.21.15.69:0/199530266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75ac0ff080 0x7f75ac19bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.268+0000 7f75b08a5700 1 -- 172.21.15.69:0/199530266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75ac105130 msgr2=0x7f75ac195700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:30.722 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.268+0000 7f75b08a5700 1 --2- 172.21.15.69:0/199530266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75ac105130 0x7f75ac195700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.268+0000 7f75b08a5700 1 -- 172.21.15.69:0/199530266 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7598009520 con 0x7f75ac0fe6d0 2023-07-12T15:28:30.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.268+0000 7f75b08a5700 1 --2- 172.21.15.69:0/199530266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75ac0fe6d0 0x7f75ac19b660 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f75a0005910 tx=0x7f75a0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:30.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.268+0000 7f75a9ffb700 1 -- 172.21.15.69:0/199530266 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a001d070 con 0x7f75ac0fe6d0 2023-07-12T15:28:30.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.268+0000 7f75a9ffb700 1 -- 172.21.15.69:0/199530266 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f75a0021410 con 0x7f75ac0fe6d0 2023-07-12T15:28:30.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.268+0000 7f75b2b09700 1 -- 172.21.15.69:0/199530266 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a00095f0 con 0x7f75ac0fe6d0 2023-07-12T15:28:30.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.268+0000 7f75a9ffb700 1 -- 172.21.15.69:0/199530266 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a0017650 con 0x7f75ac0fe6d0 2023-07-12T15:28:30.723 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.268+0000 7f75b2b09700 1 -- 172.21.15.69:0/199530266 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ac0ff7e0 con 0x7f75ac0fe6d0 2023-07-12T15:28:30.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.269+0000 7f75b2b09700 1 -- 172.21.15.69:0/199530266 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ac1965e0 con 0x7f75ac0fe6d0 2023-07-12T15:28:30.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.270+0000 7f75a9ffb700 1 -- 172.21.15.69:0/199530266 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f75a000fb10 con 0x7f75ac0fe6d0 2023-07-12T15:28:30.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.270+0000 7f75a9ffb700 1 --2- 172.21.15.69:0/199530266 >> v2:172.21.15.69:6800/1242716914 conn(0x7f759c06c810 0x7f759c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:30.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.270+0000 7f75a9ffb700 1 -- 172.21.15.69:0/199530266 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f75a008d100 con 0x7f75ac0fe6d0 2023-07-12T15:28:30.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.270+0000 7f75abfff700 1 --2- 172.21.15.69:0/199530266 >> v2:172.21.15.69:6800/1242716914 conn(0x7f759c06c810 0x7f759c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:30.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.271+0000 7f75abfff700 1 --2- 172.21.15.69:0/199530266 >> v2:172.21.15.69:6800/1242716914 conn(0x7f759c06c810 0x7f759c06ecc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f75ac196ef0 tx=0x7f7598012040 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:30.724 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.275+0000 7f75a9ffb700 1 -- 172.21.15.69:0/199530266 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f75a005afb0 con 0x7f75ac0fe6d0 2023-07-12T15:28:30.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.492+0000 7f75b2b09700 1 -- 172.21.15.69:0/199530266 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f75ac196c30 con 0x7f75ac0fe6d0 2023-07-12T15:28:30.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.492+0000 7f75a9ffb700 1 -- 172.21.15.69:0/199530266 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f75a005ab40 con 0x7f75ac0fe6d0 2023-07-12T15:28:30.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.495+0000 7f75b2b09700 1 -- 172.21.15.69:0/199530266 >> v2:172.21.15.69:6800/1242716914 conn(0x7f759c06c810 msgr2=0x7f759c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:30.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.495+0000 7f75b2b09700 1 --2- 172.21.15.69:0/199530266 >> v2:172.21.15.69:6800/1242716914 conn(0x7f759c06c810 0x7f759c06ecc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f75ac196ef0 tx=0x7f7598012040 comp rx=0 tx=0).stop 2023-07-12T15:28:30.725 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.495+0000 7f75b2b09700 1 -- 172.21.15.69:0/199530266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75ac0fe6d0 msgr2=0x7f75ac19b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:30.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.495+0000 7f75b2b09700 1 --2- 172.21.15.69:0/199530266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75ac0fe6d0 0x7f75ac19b660 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f75a0005910 tx=0x7f75a0009580 comp rx=0 tx=0).stop 2023-07-12T15:28:30.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.495+0000 7f75b2b09700 1 -- 172.21.15.69:0/199530266 shutdown_connections 2023-07-12T15:28:30.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.495+0000 7f75b2b09700 1 --2- 172.21.15.69:0/199530266 >> v2:172.21.15.69:6800/1242716914 conn(0x7f759c06c810 0x7f759c06ecc0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.495+0000 7f75b2b09700 1 --2- 172.21.15.69:0/199530266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75ac105130 0x7f75ac195700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.495+0000 7f75b2b09700 1 --2- 172.21.15.69:0/199530266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75ac0fe6d0 0x7f75ac19b660 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.495+0000 7f75b2b09700 1 --2- 172.21.15.69:0/199530266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75ac0ff080 0x7f75ac19bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.726 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.495+0000 7f75b2b09700 1 -- 172.21.15.69:0/199530266 >> 172.21.15.69:0/199530266 conn(0x7f75ac075740 msgr2=0x7f75ac0fc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:30.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.495+0000 7f75b2b09700 1 -- 172.21.15.69:0/199530266 shutdown_connections 2023-07-12T15:28:30.727 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.495+0000 7f75b2b09700 1 -- 172.21.15.69:0/199530266 wait complete. 2023-07-12T15:28:30.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:30 smithi145 ceph-mon[95801]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:30.983 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.774+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/774770279 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a941003e0 msgr2=0x7f3a94103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:30.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.774+0000 7f3a9b7ab700 1 --2- 172.21.15.69:0/774770279 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a941003e0 0x7f3a94103ce0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3a840099c0 tx=0x7f3a84009cd0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.774+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/774770279 shutdown_connections 2023-07-12T15:28:30.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.774+0000 7f3a9b7ab700 1 --2- 172.21.15.69:0/774770279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a94104410 0x7f3a9410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.774+0000 7f3a9b7ab700 1 --2- 172.21.15.69:0/774770279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a940ffa40 0x7f3a940ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.774+0000 7f3a9b7ab700 1 --2- 172.21.15.69:0/774770279 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a941003e0 0x7f3a94103ce0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.984 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.774+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/774770279 >> 172.21.15.69:0/774770279 conn(0x7f3a94075740 msgr2=0x7f3a94075b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:30.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.774+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/774770279 shutdown_connections 2023-07-12T15:28:30.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.774+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/774770279 wait complete. 2023-07-12T15:28:30.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.774+0000 7f3a9b7ab700 1 Processor -- start 2023-07-12T15:28:30.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a9b7ab700 1 -- start start 2023-07-12T15:28:30.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a9b7ab700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a940ffa40 0x7f3a94106070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:30.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a9b7ab700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a941003e0 0x7f3a941065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:30.985 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a9b7ab700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a94104410 0x7f3a94106af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:30.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a9b7ab700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a941a3da0 con 0x7f3a94104410 2023-07-12T15:28:30.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a9b7ab700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a941a3f10 con 0x7f3a941003e0 2023-07-12T15:28:30.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a9b7ab700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a941a40d0 con 0x7f3a940ffa40 2023-07-12T15:28:30.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a99d48700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a94104410 0x7f3a94106af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:30.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a99547700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a940ffa40 0x7f3a94106070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:30.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a99d48700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a94104410 0x7f3a94106af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41426/0 (socket says 172.21.15.69:41426) 2023-07-12T15:28:30.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a99d48700 1 -- 172.21.15.69:0/1626286909 learned_addr learned my addr 172.21.15.69:0/1626286909 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:30.986 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a98d46700 1 --2- 172.21.15.69:0/1626286909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a941003e0 0x7f3a941065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:30.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a99d48700 1 -- 172.21.15.69:0/1626286909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a940ffa40 msgr2=0x7f3a94106070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:30.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a99d48700 1 --2- 172.21.15.69:0/1626286909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a940ffa40 0x7f3a94106070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a99d48700 1 -- 172.21.15.69:0/1626286909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a941003e0 msgr2=0x7f3a941065b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:30.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a99d48700 1 --2- 172.21.15.69:0/1626286909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a941003e0 0x7f3a941065b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:30.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a99d48700 1 -- 172.21.15.69:0/1626286909 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a90009520 con 0x7f3a94104410 2023-07-12T15:28:30.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a98d46700 1 --2- 172.21.15.69:0/1626286909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a941003e0 0x7f3a941065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:28:30.987 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a99547700 1 --2- 172.21.15.69:0/1626286909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a940ffa40 0x7f3a94106070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:28:30.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.775+0000 7f3a99d48700 1 --2- 172.21.15.69:0/1626286909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a94104410 0x7f3a94106af0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c007ee0 tx=0x7f3a8c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:30.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.776+0000 7f3a8a7fc700 1 -- 172.21.15.69:0/1626286909 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c017070 con 0x7f3a94104410 2023-07-12T15:28:30.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.776+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/1626286909 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a840095f0 con 0x7f3a94104410 2023-07-12T15:28:30.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.776+0000 7f3a8a7fc700 1 -- 172.21.15.69:0/1626286909 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f3a8c00f040 con 0x7f3a94104410 2023-07-12T15:28:30.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.776+0000 7f3a8a7fc700 1 -- 172.21.15.69:0/1626286909 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c005710 con 0x7f3a94104410 2023-07-12T15:28:30.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.776+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/1626286909 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a941a46f0 con 0x7f3a94104410 2023-07-12T15:28:30.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.777+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/1626286909 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a9404e800 con 0x7f3a94104410 2023-07-12T15:28:30.988 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.777+0000 7f3a8a7fc700 1 -- 172.21.15.69:0/1626286909 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f3a8c005870 con 0x7f3a94104410 2023-07-12T15:28:30.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.778+0000 7f3a8a7fc700 1 --2- 172.21.15.69:0/1626286909 >> v2:172.21.15.69:6800/1242716914 conn(0x7f3a8006c810 0x7f3a8006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:30.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.778+0000 7f3a8a7fc700 1 -- 172.21.15.69:0/1626286909 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f3a8c08b260 con 0x7f3a94104410 2023-07-12T15:28:30.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.778+0000 7f3a99547700 1 --2- 172.21.15.69:0/1626286909 >> v2:172.21.15.69:6800/1242716914 conn(0x7f3a8006c810 0x7f3a8006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:30.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.778+0000 7f3a99547700 1 --2- 172.21.15.69:0/1626286909 >> v2:172.21.15.69:6800/1242716914 conn(0x7f3a8006c810 0x7f3a8006ecc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3a90009fd0 tx=0x7f3a90009260 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:30.989 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.782+0000 7f3a8a7fc700 1 -- 172.21.15.69:0/1626286909 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3a8c059110 con 0x7f3a94104410 2023-07-12T15:28:30.997 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.994+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/1626286909 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f3a941a4910 con 0x7f3a94104410 2023-07-12T15:28:30.997 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.994+0000 7f3a8a7fc700 1 -- 172.21.15.69:0/1626286909 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3a8c058ca0 con 0x7f3a94104410 2023-07-12T15:28:30.997 INFO:teuthology.orchestra.run.smithi069.stdout:231928233997 2023-07-12T15:28:30.999 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.997+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/1626286909 >> v2:172.21.15.69:6800/1242716914 conn(0x7f3a8006c810 msgr2=0x7f3a8006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:30.999 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.997+0000 7f3a9b7ab700 1 --2- 172.21.15.69:0/1626286909 >> v2:172.21.15.69:6800/1242716914 conn(0x7f3a8006c810 0x7f3a8006ecc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3a90009fd0 tx=0x7f3a90009260 comp rx=0 tx=0).stop 2023-07-12T15:28:30.999 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.997+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/1626286909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a94104410 msgr2=0x7f3a94106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:31.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.997+0000 7f3a9b7ab700 1 --2- 172.21.15.69:0/1626286909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a94104410 0x7f3a94106af0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c007ee0 tx=0x7f3a8c00d3b0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.997+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/1626286909 shutdown_connections 2023-07-12T15:28:31.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.997+0000 7f3a9b7ab700 1 --2- 172.21.15.69:0/1626286909 >> v2:172.21.15.69:6800/1242716914 conn(0x7f3a8006c810 0x7f3a8006ecc0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.997+0000 7f3a9b7ab700 1 --2- 172.21.15.69:0/1626286909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a94104410 0x7f3a94106af0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.997+0000 7f3a9b7ab700 1 --2- 172.21.15.69:0/1626286909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a940ffa40 0x7f3a94106070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.997+0000 7f3a9b7ab700 1 --2- 172.21.15.69:0/1626286909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a941003e0 0x7f3a941065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.000 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.997+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/1626286909 >> 172.21.15.69:0/1626286909 conn(0x7f3a94075740 msgr2=0x7f3a94102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:31.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.997+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/1626286909 shutdown_connections 2023-07-12T15:28:31.001 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:30.997+0000 7f3a9b7ab700 1 -- 172.21.15.69:0/1626286909 wait complete. 2023-07-12T15:28:31.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.174+0000 7f38ca307700 1 -- 172.21.15.69:0/1161640475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38c4100c60 msgr2=0x7f38c41010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:31.259 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.174+0000 7f38ca307700 1 --2- 172.21.15.69:0/1161640475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38c4100c60 0x7f38c41010d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f38b80099c0 tx=0x7f38b8009cd0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.174+0000 7f38ca307700 1 -- 172.21.15.69:0/1161640475 shutdown_connections 2023-07-12T15:28:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.174+0000 7f38ca307700 1 --2- 172.21.15.69:0/1161640475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c4101610 0x7f38c410c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.174+0000 7f38ca307700 1 --2- 172.21.15.69:0/1161640475 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38c40ff400 0x7f38c40ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.174+0000 7f38ca307700 1 --2- 172.21.15.69:0/1161640475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38c4100c60 0x7f38c41010d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.174+0000 7f38ca307700 1 -- 172.21.15.69:0/1161640475 >> 172.21.15.69:0/1161640475 conn(0x7f38c4076150 msgr2=0x7f38c4076550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.174+0000 7f38ca307700 1 -- 172.21.15.69:0/1161640475 shutdown_connections 2023-07-12T15:28:31.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.174+0000 7f38ca307700 1 -- 172.21.15.69:0/1161640475 wait complete. 2023-07-12T15:28:31.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.174+0000 7f38ca307700 1 Processor -- start 2023-07-12T15:28:31.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38ca307700 1 -- start start 2023-07-12T15:28:31.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38ca307700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c40ff400 0x7f38c419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:31.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38ca307700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38c4100c60 0x7f38c419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:31.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38ca307700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38c4101610 0x7f38c4197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:31.261 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38ca307700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c4198150 con 0x7f38c40ff400 2023-07-12T15:28:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38ca307700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c41982c0 con 0x7f38c4100c60 2023-07-12T15:28:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38ca307700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c4198480 con 0x7f38c4101610 2023-07-12T15:28:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38c3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c40ff400 0x7f38c419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38c88a4700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38c4101610 0x7f38c4197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38c3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c40ff400 0x7f38c419da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41436/0 (socket says 172.21.15.69:41436) 2023-07-12T15:28:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38c3fff700 1 -- 172.21.15.69:0/3980435047 learned_addr learned my addr 172.21.15.69:0/3980435047 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38c37fe700 1 --2- 172.21.15.69:0/3980435047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38c4100c60 0x7f38c419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38c3fff700 1 -- 172.21.15.69:0/3980435047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38c4101610 msgr2=0x7f38c4197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:31.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38c3fff700 1 --2- 172.21.15.69:0/3980435047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38c4101610 0x7f38c4197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38c3fff700 1 -- 172.21.15.69:0/3980435047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38c4100c60 msgr2=0x7f38c419dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:31.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38c3fff700 1 --2- 172.21.15.69:0/3980435047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38c4100c60 0x7f38c419dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38c3fff700 1 -- 172.21.15.69:0/3980435047 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b80095f0 con 0x7f38c40ff400 2023-07-12T15:28:31.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38c88a4700 1 --2- 172.21.15.69:0/3980435047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38c4101610 0x7f38c4197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:28:31.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38c37fe700 1 --2- 172.21.15.69:0/3980435047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38c4100c60 0x7f38c419dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:28:31.263 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.175+0000 7f38c3fff700 1 --2- 172.21.15.69:0/3980435047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c40ff400 0x7f38c419da60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f38b400b530 tx=0x7f38b400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:31.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.176+0000 7f38c17fa700 1 -- 172.21.15.69:0/3980435047 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4011830 con 0x7f38c40ff400 2023-07-12T15:28:31.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.176+0000 7f38ca307700 1 -- 172.21.15.69:0/3980435047 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c4198760 con 0x7f38c40ff400 2023-07-12T15:28:31.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.176+0000 7f38c17fa700 1 -- 172.21.15.69:0/3980435047 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f38b4011e20 con 0x7f38c40ff400 2023-07-12T15:28:31.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.176+0000 7f38c17fa700 1 -- 172.21.15.69:0/3980435047 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4010e80 con 0x7f38c40ff400 2023-07-12T15:28:31.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.176+0000 7f38ca307700 1 -- 172.21.15.69:0/3980435047 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c41a4ad0 con 0x7f38c40ff400 2023-07-12T15:28:31.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.177+0000 7f38ca307700 1 -- 172.21.15.69:0/3980435047 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c404e800 con 0x7f38c40ff400 2023-07-12T15:28:31.264 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.177+0000 7f38c17fa700 1 -- 172.21.15.69:0/3980435047 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f38b401a420 con 0x7f38c40ff400 2023-07-12T15:28:31.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.178+0000 7f38c17fa700 1 --2- 172.21.15.69:0/3980435047 >> v2:172.21.15.69:6800/1242716914 conn(0x7f38ac06c810 0x7f38ac06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:31.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.178+0000 7f38c37fe700 1 --2- 172.21.15.69:0/3980435047 >> v2:172.21.15.69:6800/1242716914 conn(0x7f38ac06c810 0x7f38ac06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:31.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.178+0000 7f38c17fa700 1 -- 172.21.15.69:0/3980435047 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f38b408b690 con 0x7f38c40ff400 2023-07-12T15:28:31.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.178+0000 7f38c37fe700 1 --2- 172.21.15.69:0/3980435047 >> v2:172.21.15.69:6800/1242716914 conn(0x7f38ac06c810 0x7f38ac06ecc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f38b800b580 tx=0x7f38b8005fb0 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:31.265 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.182+0000 7f38c17fa700 1 -- 172.21.15.69:0/3980435047 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f38b4059540 con 0x7f38c40ff400 2023-07-12T15:28:31.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.381+0000 7f38ca307700 1 -- 172.21.15.69:0/3980435047 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f38c41a4d20 con 0x7f38c40ff400 2023-07-12T15:28:31.384 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.381+0000 7f38c17fa700 1 -- 172.21.15.69:0/3980435047 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f38b40590d0 con 0x7f38c40ff400 2023-07-12T15:28:31.387 INFO:teuthology.orchestra.run.smithi069.stdout:77309411364 2023-07-12T15:28:31.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.384+0000 7f38ca307700 1 -- 172.21.15.69:0/3980435047 >> v2:172.21.15.69:6800/1242716914 conn(0x7f38ac06c810 msgr2=0x7f38ac06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:31.387 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.384+0000 7f38ca307700 1 --2- 172.21.15.69:0/3980435047 >> v2:172.21.15.69:6800/1242716914 conn(0x7f38ac06c810 0x7f38ac06ecc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f38b800b580 tx=0x7f38b8005fb0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.384+0000 7f38ca307700 1 -- 172.21.15.69:0/3980435047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c40ff400 msgr2=0x7f38c419da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:31.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.384+0000 7f38ca307700 1 --2- 172.21.15.69:0/3980435047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c40ff400 0x7f38c419da60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f38b400b530 tx=0x7f38b400b840 comp rx=0 tx=0).stop 2023-07-12T15:28:31.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.384+0000 7f38ca307700 1 -- 172.21.15.69:0/3980435047 shutdown_connections 2023-07-12T15:28:31.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.384+0000 7f38ca307700 1 --2- 172.21.15.69:0/3980435047 >> v2:172.21.15.69:6800/1242716914 conn(0x7f38ac06c810 0x7f38ac06ecc0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.384+0000 7f38ca307700 1 --2- 172.21.15.69:0/3980435047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38c4101610 0x7f38c4197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.384+0000 7f38ca307700 1 --2- 172.21.15.69:0/3980435047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c40ff400 0x7f38c419da60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.388 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.384+0000 7f38ca307700 1 --2- 172.21.15.69:0/3980435047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38c4100c60 0x7f38c419dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:31.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.384+0000 7f38ca307700 1 -- 172.21.15.69:0/3980435047 >> 172.21.15.69:0/3980435047 conn(0x7f38c4076150 msgr2=0x7f38c4068e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:31.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.384+0000 7f38ca307700 1 -- 172.21.15.69:0/3980435047 shutdown_connections 2023-07-12T15:28:31.389 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:31.384+0000 7f38ca307700 1 -- 172.21.15.69:0/3980435047 wait complete. 2023-07-12T15:28:31.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/199530266' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-12T15:28:31.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1626286909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-12T15:28:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3980435047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-12T15:28:31.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/199530266' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-12T15:28:31.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1626286909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-12T15:28:31.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3980435047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-12T15:28:31.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/199530266' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-12T15:28:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1626286909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-12T15:28:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3980435047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-12T15:28:32.121 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574887 got 55834574889 for osd.1 2023-07-12T15:28:32.121 DEBUG:teuthology.parallel:result is None 2023-07-12T15:28:32.237 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233996 got 231928233997 for osd.7 2023-07-12T15:28:32.237 DEBUG:teuthology.parallel:result is None 2023-07-12T15:28:32.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.524+0000 7fa6ce287700 1 -- 172.21.15.69:0/1893837709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c8101d80 msgr2=0x7fa6c810a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.524+0000 7fa6ce287700 1 --2- 172.21.15.69:0/1893837709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c8101d80 0x7fa6c810a1f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009990 tx=0x7fa6bc009ca0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.524+0000 7fa6ce287700 1 -- 172.21.15.69:0/1893837709 shutdown_connections 2023-07-12T15:28:32.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.524+0000 7fa6ce287700 1 --2- 172.21.15.69:0/1893837709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6c810a860 0x7fa6c810ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.527 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.524+0000 7fa6ce287700 1 --2- 172.21.15.69:0/1893837709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c81013e0 0x7fa6c81017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.524+0000 7fa6ce287700 1 --2- 172.21.15.69:0/1893837709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c8101d80 0x7fa6c810a1f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.528 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.524+0000 7fa6ce287700 1 -- 172.21.15.69:0/1893837709 >> 172.21.15.69:0/1893837709 conn(0x7fa6c8076150 msgr2=0x7fa6c8076550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:32.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.524+0000 7fa6ce287700 1 -- 172.21.15.69:0/1893837709 shutdown_connections 2023-07-12T15:28:32.529 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.524+0000 7fa6ce287700 1 -- 172.21.15.69:0/1893837709 wait complete. 2023-07-12T15:28:32.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.525+0000 7fa6ce287700 1 Processor -- start 2023-07-12T15:28:32.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.525+0000 7fa6ce287700 1 -- start start 2023-07-12T15:28:32.531 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.525+0000 7fa6ce287700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c81013e0 0x7fa6c819dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:32.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.525+0000 7fa6ce287700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c8101d80 0x7fa6c819dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:32.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.525+0000 7fa6ce287700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6c810a860 0x7fa6c8197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:32.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.525+0000 7fa6ce287700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c81981a0 con 0x7fa6c8101d80 2023-07-12T15:28:32.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.525+0000 7fa6ce287700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c8198310 con 0x7fa6c810a860 2023-07-12T15:28:32.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.525+0000 7fa6ce287700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c81984d0 con 0x7fa6c81013e0 2023-07-12T15:28:32.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c7fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c81013e0 0x7fa6c819dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:32.532 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c8101d80 0x7fa6c819dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:32.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c7fff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c81013e0 0x7fa6c819dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34874/0 (socket says 172.21.15.69:34874) 2023-07-12T15:28:32.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c8101d80 0x7fa6c819dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41448/0 (socket says 172.21.15.69:41448) 2023-07-12T15:28:32.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c7fff700 1 -- 172.21.15.69:0/869618262 learned_addr learned my addr 172.21.15.69:0/869618262 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:32.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6cc824700 1 --2- 172.21.15.69:0/869618262 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6c810a860 0x7fa6c8197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:32.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c7fff700 1 -- 172.21.15.69:0/869618262 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6c810a860 msgr2=0x7fa6c8197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c7fff700 1 --2- 172.21.15.69:0/869618262 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6c810a860 0x7fa6c8197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c7fff700 1 -- 172.21.15.69:0/869618262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c8101d80 msgr2=0x7fa6c819dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.533 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c7fff700 1 --2- 172.21.15.69:0/869618262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c8101d80 0x7fa6c819dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c7fff700 1 -- 172.21.15.69:0/869618262 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b8009520 con 0x7fa6c81013e0 2023-07-12T15:28:32.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c77fe700 1 --2- 172.21.15.69:0/869618262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c8101d80 0x7fa6c819dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:28:32.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c7fff700 1 --2- 172.21.15.69:0/869618262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c81013e0 0x7fa6c819dab0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa6b800eb30 tx=0x7fa6b800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:32.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c57fa700 1 -- 172.21.15.69:0/869618262 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8009880 con 0x7fa6c81013e0 2023-07-12T15:28:32.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c57fa700 1 -- 172.21.15.69:0/869618262 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fa6b80044b0 con 0x7fa6c81013e0 2023-07-12T15:28:32.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6ce287700 1 -- 172.21.15.69:0/869618262 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6bc0095f0 con 0x7fa6c81013e0 2023-07-12T15:28:32.534 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6c57fa700 1 -- 172.21.15.69:0/869618262 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8003ca0 con 0x7fa6c81013e0 2023-07-12T15:28:32.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.526+0000 7fa6ce287700 1 -- 172.21.15.69:0/869618262 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c81a48d0 con 0x7fa6c81013e0 2023-07-12T15:28:32.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.527+0000 7fa6ce287700 1 -- 172.21.15.69:0/869618262 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c8107d20 con 0x7fa6c81013e0 2023-07-12T15:28:32.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.528+0000 7fa6c57fa700 1 -- 172.21.15.69:0/869618262 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7fa6b8003e00 con 0x7fa6c81013e0 2023-07-12T15:28:32.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.528+0000 7fa6c57fa700 1 --2- 172.21.15.69:0/869618262 >> v2:172.21.15.69:6800/1242716914 conn(0x7fa6b006c810 0x7fa6b006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:32.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.528+0000 7fa6c57fa700 1 -- 172.21.15.69:0/869618262 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fa6b8014070 con 0x7fa6c81013e0 2023-07-12T15:28:32.535 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.528+0000 7fa6c77fe700 1 --2- 172.21.15.69:0/869618262 >> v2:172.21.15.69:6800/1242716914 conn(0x7fa6b006c810 0x7fa6b006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:32.536 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.529+0000 7fa6c77fe700 1 --2- 172.21.15.69:0/869618262 >> v2:172.21.15.69:6800/1242716914 conn(0x7fa6b006c810 0x7fa6b006ecc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00b650 tx=0x7fa6bc005fb0 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:32.540 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.537+0000 7fa6c57fa700 1 -- 172.21.15.69:0/869618262 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa6b8059f90 con 0x7fa6c81013e0 2023-07-12T15:28:32.589 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.586+0000 7f7e008ce700 1 -- 172.21.15.69:0/3838243602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dfc101610 msgr2=0x7f7dfc10c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.586+0000 7f7e008ce700 1 --2- 172.21.15.69:0/3838243602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dfc101610 0x7f7dfc10c000 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009fc0 tx=0x7f7df400b330 comp rx=0 tx=0).stop 2023-07-12T15:28:32.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.586+0000 7f7e008ce700 1 -- 172.21.15.69:0/3838243602 shutdown_connections 2023-07-12T15:28:32.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.586+0000 7f7e008ce700 1 --2- 172.21.15.69:0/3838243602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dfc101610 0x7f7dfc10c000 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.586+0000 7f7e008ce700 1 --2- 172.21.15.69:0/3838243602 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dfc0ff400 0x7f7dfc0ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.590 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.586+0000 7f7e008ce700 1 --2- 172.21.15.69:0/3838243602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dfc100c60 0x7f7dfc1010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.586+0000 7f7e008ce700 1 -- 172.21.15.69:0/3838243602 >> 172.21.15.69:0/3838243602 conn(0x7f7dfc076150 msgr2=0x7f7dfc076550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:32.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.586+0000 7f7e008ce700 1 -- 172.21.15.69:0/3838243602 shutdown_connections 2023-07-12T15:28:32.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.586+0000 7f7e008ce700 1 -- 172.21.15.69:0/3838243602 wait complete. 2023-07-12T15:28:32.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.587+0000 7f7e008ce700 1 Processor -- start 2023-07-12T15:28:32.591 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.587+0000 7f7e008ce700 1 -- start start 2023-07-12T15:28:32.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.587+0000 7f7e008ce700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dfc0ff400 0x7f7dfc19b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:32.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.587+0000 7f7e008ce700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dfc100c60 0x7f7dfc19be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:32.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.587+0000 7f7e008ce700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dfc101610 0x7f7dfc195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:32.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.587+0000 7f7e008ce700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc195fb0 con 0x7f7dfc0ff400 2023-07-12T15:28:32.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.587+0000 7f7e008ce700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc196120 con 0x7f7dfc101610 2023-07-12T15:28:32.592 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.587+0000 7f7e008ce700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc1962e0 con 0x7f7dfc100c60 2023-07-12T15:28:32.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7dfa59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dfc0ff400 0x7f7dfc19b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:32.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7dfa59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dfc0ff400 0x7f7dfc19b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41460/0 (socket says 172.21.15.69:41460) 2023-07-12T15:28:32.594 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7dfa59c700 1 -- 172.21.15.69:0/3237100334 learned_addr learned my addr 172.21.15.69:0/3237100334 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:32.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7df9d9b700 1 --2- 172.21.15.69:0/3237100334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dfc100c60 0x7f7dfc19be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:32.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7dfad9d700 1 --2- 172.21.15.69:0/3237100334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dfc101610 0x7f7dfc195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:32.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7dfa59c700 1 -- 172.21.15.69:0/3237100334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dfc100c60 msgr2=0x7f7dfc19be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7dfa59c700 1 --2- 172.21.15.69:0/3237100334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dfc100c60 0x7f7dfc19be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7dfa59c700 1 -- 172.21.15.69:0/3237100334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dfc101610 msgr2=0x7f7dfc195940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7dfa59c700 1 --2- 172.21.15.69:0/3237100334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dfc101610 0x7f7dfc195940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.595 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7dfa59c700 1 -- 172.21.15.69:0/3237100334 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df4009c40 con 0x7f7dfc0ff400 2023-07-12T15:28:32.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7dfad9d700 1 --2- 172.21.15.69:0/3237100334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dfc101610 0x7f7dfc195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:28:32.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7dfa59c700 1 --2- 172.21.15.69:0/3237100334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dfc0ff400 0x7f7dfc19b8c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00e9a0 tx=0x7f7dec00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:32.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7deb7fe700 1 -- 172.21.15.69:0/3237100334 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec009790 con 0x7f7dfc0ff400 2023-07-12T15:28:32.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7e008ce700 1 -- 172.21.15.69:0/3237100334 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dfc1965c0 con 0x7f7dfc0ff400 2023-07-12T15:28:32.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7deb7fe700 1 -- 172.21.15.69:0/3237100334 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f7dec00cc50 con 0x7f7dfc0ff400 2023-07-12T15:28:32.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.588+0000 7f7deb7fe700 1 -- 172.21.15.69:0/3237100334 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec00eee0 con 0x7f7dfc0ff400 2023-07-12T15:28:32.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.589+0000 7f7e008ce700 1 -- 172.21.15.69:0/3237100334 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dfc10ccc0 con 0x7f7dfc0ff400 2023-07-12T15:28:32.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.590+0000 7f7e008ce700 1 -- 172.21.15.69:0/3237100334 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dfc04e800 con 0x7f7dfc0ff400 2023-07-12T15:28:32.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.590+0000 7f7deb7fe700 1 -- 172.21.15.69:0/3237100334 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f7dec00c7c0 con 0x7f7dfc0ff400 2023-07-12T15:28:32.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.591+0000 7f7deb7fe700 1 --2- 172.21.15.69:0/3237100334 >> v2:172.21.15.69:6800/1242716914 conn(0x7f7de406c810 0x7f7de406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:32.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.591+0000 7f7df9d9b700 1 --2- 172.21.15.69:0/3237100334 >> v2:172.21.15.69:6800/1242716914 conn(0x7f7de406c810 0x7f7de406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:32.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.591+0000 7f7deb7fe700 1 -- 172.21.15.69:0/3237100334 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f7dec094290 con 0x7f7dfc0ff400 2023-07-12T15:28:32.597 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.591+0000 7f7df9d9b700 1 --2- 172.21.15.69:0/3237100334 >> v2:172.21.15.69:6800/1242716914 conn(0x7f7de406c810 0x7f7de406ecc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc105c40 tx=0x7f7df0009310 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:32.599 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.597+0000 7f7deb7fe700 1 -- 172.21.15.69:0/3237100334 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7dec059720 con 0x7f7dfc0ff400 2023-07-12T15:28:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:32 smithi069 ceph-mon[102777]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:32.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:32 smithi069 ceph-mon[94241]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:32.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.741+0000 7fa6ce287700 1 -- 172.21.15.69:0/869618262 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fa6c804e800 con 0x7fa6c81013e0 2023-07-12T15:28:32.744 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.741+0000 7fa6c57fa700 1 -- 172.21.15.69:0/869618262 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa6b8059b20 con 0x7fa6c81013e0 2023-07-12T15:28:32.745 INFO:teuthology.orchestra.run.smithi069.stdout:163208757271 2023-07-12T15:28:32.746 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.743+0000 7fa6ce287700 1 -- 172.21.15.69:0/869618262 >> v2:172.21.15.69:6800/1242716914 conn(0x7fa6b006c810 msgr2=0x7fa6b006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.743+0000 7fa6ce287700 1 --2- 172.21.15.69:0/869618262 >> v2:172.21.15.69:6800/1242716914 conn(0x7fa6b006c810 0x7fa6b006ecc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00b650 tx=0x7fa6bc005fb0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.743+0000 7fa6ce287700 1 -- 172.21.15.69:0/869618262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c81013e0 msgr2=0x7fa6c819dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.743+0000 7fa6ce287700 1 --2- 172.21.15.69:0/869618262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c81013e0 0x7fa6c819dab0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa6b800eb30 tx=0x7fa6b800ee40 comp rx=0 tx=0).stop 2023-07-12T15:28:32.747 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.744+0000 7fa6ce287700 1 -- 172.21.15.69:0/869618262 shutdown_connections 2023-07-12T15:28:32.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.744+0000 7fa6ce287700 1 --2- 172.21.15.69:0/869618262 >> v2:172.21.15.69:6800/1242716914 conn(0x7fa6b006c810 0x7fa6b006ecc0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.744+0000 7fa6ce287700 1 --2- 172.21.15.69:0/869618262 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6c810a860 0x7fa6c8197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.748 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.744+0000 7fa6ce287700 1 --2- 172.21.15.69:0/869618262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c81013e0 0x7fa6c819dab0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.744+0000 7fa6ce287700 1 --2- 172.21.15.69:0/869618262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c8101d80 0x7fa6c819dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.744+0000 7fa6ce287700 1 -- 172.21.15.69:0/869618262 >> 172.21.15.69:0/869618262 conn(0x7fa6c8076150 msgr2=0x7fa6c80ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:32.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.744+0000 7fa6ce287700 1 -- 172.21.15.69:0/869618262 shutdown_connections 2023-07-12T15:28:32.749 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.744+0000 7fa6ce287700 1 -- 172.21.15.69:0/869618262 wait complete. 2023-07-12T15:28:32.803 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.801+0000 7f7e008ce700 1 -- 172.21.15.69:0/3237100334 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7dfc196e50 con 0x7f7dfc0ff400 2023-07-12T15:28:32.804 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.802+0000 7f7deb7fe700 1 -- 172.21.15.69:0/3237100334 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7dec0621f0 con 0x7f7dfc0ff400 2023-07-12T15:28:32.807 INFO:teuthology.orchestra.run.smithi069.stdout:34359738413 2023-07-12T15:28:32.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.806+0000 7f7e008ce700 1 -- 172.21.15.69:0/3237100334 >> v2:172.21.15.69:6800/1242716914 conn(0x7f7de406c810 msgr2=0x7f7de406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.806+0000 7f7e008ce700 1 --2- 172.21.15.69:0/3237100334 >> v2:172.21.15.69:6800/1242716914 conn(0x7f7de406c810 0x7f7de406ecc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc105c40 tx=0x7f7df0009310 comp rx=0 tx=0).stop 2023-07-12T15:28:32.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.806+0000 7f7e008ce700 1 -- 172.21.15.69:0/3237100334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dfc0ff400 msgr2=0x7f7dfc19b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.808 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.806+0000 7f7e008ce700 1 --2- 172.21.15.69:0/3237100334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dfc0ff400 0x7f7dfc19b8c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00e9a0 tx=0x7f7dec00ed60 comp rx=0 tx=0).stop 2023-07-12T15:28:32.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.806+0000 7f7e008ce700 1 -- 172.21.15.69:0/3237100334 shutdown_connections 2023-07-12T15:28:32.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.806+0000 7f7e008ce700 1 --2- 172.21.15.69:0/3237100334 >> v2:172.21.15.69:6800/1242716914 conn(0x7f7de406c810 0x7f7de406ecc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.806+0000 7f7e008ce700 1 --2- 172.21.15.69:0/3237100334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dfc101610 0x7f7dfc195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.809 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.806+0000 7f7e008ce700 1 --2- 172.21.15.69:0/3237100334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dfc0ff400 0x7f7dfc19b8c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.806+0000 7f7e008ce700 1 --2- 172.21.15.69:0/3237100334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dfc100c60 0x7f7dfc19be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.806+0000 7f7e008ce700 1 -- 172.21.15.69:0/3237100334 >> 172.21.15.69:0/3237100334 conn(0x7f7dfc076150 msgr2=0x7f7dfc0fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:32.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.806+0000 7f7e008ce700 1 -- 172.21.15.69:0/3237100334 shutdown_connections 2023-07-12T15:28:32.810 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.806+0000 7f7e008ce700 1 -- 172.21.15.69:0/3237100334 wait complete. 2023-07-12T15:28:32.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:32 smithi145 ceph-mon[95801]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:32.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.691+0000 7efea3e0a700 1 -- 172.21.15.69:0/294622746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9c101cf0 msgr2=0x7efe9c10a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.691+0000 7efea3e0a700 1 --2- 172.21.15.69:0/294622746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9c101cf0 0x7efe9c10a1f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efe8c0098f0 tx=0x7efe8c009c00 comp rx=0 tx=0).stop 2023-07-12T15:28:32.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.691+0000 7efea3e0a700 1 -- 172.21.15.69:0/294622746 shutdown_connections 2023-07-12T15:28:32.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.691+0000 7efea3e0a700 1 --2- 172.21.15.69:0/294622746 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe9c10a730 0x7efe9c10cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.943 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.691+0000 7efea3e0a700 1 --2- 172.21.15.69:0/294622746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe9c1013e0 0x7efe9c1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.691+0000 7efea3e0a700 1 --2- 172.21.15.69:0/294622746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9c101cf0 0x7efe9c10a1f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.691+0000 7efea3e0a700 1 -- 172.21.15.69:0/294622746 >> 172.21.15.69:0/294622746 conn(0x7efe9c076160 msgr2=0x7efe9c076560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:32.944 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.691+0000 7efea3e0a700 1 -- 172.21.15.69:0/294622746 shutdown_connections 2023-07-12T15:28:32.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.691+0000 7efea3e0a700 1 -- 172.21.15.69:0/294622746 wait complete. 2023-07-12T15:28:32.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.692+0000 7efea3e0a700 1 Processor -- start 2023-07-12T15:28:32.945 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.692+0000 7efea3e0a700 1 -- start start 2023-07-12T15:28:32.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.692+0000 7efea3e0a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe9c1013e0 0x7efe9c195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:32.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea3e0a700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe9c101cf0 0x7efe9c195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:32.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea3e0a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9c10a730 0x7efe9c19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:32.946 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea3e0a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c19a920 con 0x7efe9c10a730 2023-07-12T15:28:32.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea3e0a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c19aa90 con 0x7efe9c1013e0 2023-07-12T15:28:32.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea3e0a700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c19ac50 con 0x7efe9c101cf0 2023-07-12T15:28:32.947 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea13a5700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe9c101cf0 0x7efe9c195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:32.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea13a5700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe9c101cf0 0x7efe9c195eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34904/0 (socket says 172.21.15.69:34904) 2023-07-12T15:28:32.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea13a5700 1 -- 172.21.15.69:0/872103410 learned_addr learned my addr 172.21.15.69:0/872103410 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:32.948 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea23a7700 1 --2- 172.21.15.69:0/872103410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9c10a730 0x7efe9c19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:32.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea1ba6700 1 --2- 172.21.15.69:0/872103410 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe9c1013e0 0x7efe9c195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:32.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea13a5700 1 -- 172.21.15.69:0/872103410 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe9c1013e0 msgr2=0x7efe9c195970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.949 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea13a5700 1 --2- 172.21.15.69:0/872103410 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe9c1013e0 0x7efe9c195970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea13a5700 1 -- 172.21.15.69:0/872103410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9c10a730 msgr2=0x7efe9c19a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea13a5700 1 --2- 172.21.15.69:0/872103410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9c10a730 0x7efe9c19a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea13a5700 1 -- 172.21.15.69:0/872103410 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c009520 con 0x7efe9c101cf0 2023-07-12T15:28:32.950 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea1ba6700 1 --2- 172.21.15.69:0/872103410 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe9c1013e0 0x7efe9c195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:28:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea13a5700 1 --2- 172.21.15.69:0/872103410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe9c101cf0 0x7efe9c195eb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009230 tx=0x7efe8c00b560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efe92ffd700 1 -- 172.21.15.69:0/872103410 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c01c070 con 0x7efe9c101cf0 2023-07-12T15:28:32.951 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efe92ffd700 1 -- 172.21.15.69:0/872103410 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7efe8c005040 con 0x7efe9c101cf0 2023-07-12T15:28:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea3e0a700 1 -- 172.21.15.69:0/872103410 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c19aed0 con 0x7efe9c101cf0 2023-07-12T15:28:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efe92ffd700 1 -- 172.21.15.69:0/872103410 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c0038c0 con 0x7efe9c101cf0 2023-07-12T15:28:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.693+0000 7efea3e0a700 1 -- 172.21.15.69:0/872103410 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c19b1a0 con 0x7efe9c101cf0 2023-07-12T15:28:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.695+0000 7efea3e0a700 1 -- 172.21.15.69:0/872103410 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c104fc0 con 0x7efe9c101cf0 2023-07-12T15:28:32.952 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.695+0000 7efe92ffd700 1 -- 172.21.15.69:0/872103410 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7efe8c020030 con 0x7efe9c101cf0 2023-07-12T15:28:32.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.695+0000 7efe92ffd700 1 --2- 172.21.15.69:0/872103410 >> v2:172.21.15.69:6800/1242716914 conn(0x7efe8806c810 0x7efe8806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:32.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.695+0000 7efea1ba6700 1 --2- 172.21.15.69:0/872103410 >> v2:172.21.15.69:6800/1242716914 conn(0x7efe8806c810 0x7efe8806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:32.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.695+0000 7efe92ffd700 1 -- 172.21.15.69:0/872103410 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7efe8c08ce40 con 0x7efe9c101cf0 2023-07-12T15:28:32.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.696+0000 7efea1ba6700 1 --2- 172.21.15.69:0/872103410 >> v2:172.21.15.69:6800/1242716914 conn(0x7efe8806c810 0x7efe8806ecc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe98005fd0 tx=0x7efe98009310 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:32.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.700+0000 7efe92ffd700 1 -- 172.21.15.69:0/872103410 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7efe8c05acf0 con 0x7efe9c101cf0 2023-07-12T15:28:32.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.918+0000 7efea3e0a700 1 -- 172.21.15.69:0/872103410 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7efe9c196cd0 con 0x7efe9c101cf0 2023-07-12T15:28:32.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.918+0000 7efe92ffd700 1 -- 172.21.15.69:0/872103410 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7efe8c05a880 con 0x7efe9c101cf0 2023-07-12T15:28:32.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.920+0000 7efea3e0a700 1 -- 172.21.15.69:0/872103410 >> v2:172.21.15.69:6800/1242716914 conn(0x7efe8806c810 msgr2=0x7efe8806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.920+0000 7efea3e0a700 1 --2- 172.21.15.69:0/872103410 >> v2:172.21.15.69:6800/1242716914 conn(0x7efe8806c810 0x7efe8806ecc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe98005fd0 tx=0x7efe98009310 comp rx=0 tx=0).stop 2023-07-12T15:28:32.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.920+0000 7efea3e0a700 1 -- 172.21.15.69:0/872103410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe9c101cf0 msgr2=0x7efe9c195eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.920+0000 7efea3e0a700 1 --2- 172.21.15.69:0/872103410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe9c101cf0 0x7efe9c195eb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009230 tx=0x7efe8c00b560 comp rx=0 tx=0).stop 2023-07-12T15:28:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.920+0000 7efea3e0a700 1 -- 172.21.15.69:0/872103410 shutdown_connections 2023-07-12T15:28:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.920+0000 7efea3e0a700 1 --2- 172.21.15.69:0/872103410 >> v2:172.21.15.69:6800/1242716914 conn(0x7efe8806c810 0x7efe8806ecc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.920+0000 7efea3e0a700 1 --2- 172.21.15.69:0/872103410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9c10a730 0x7efe9c19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.920+0000 7efea3e0a700 1 --2- 172.21.15.69:0/872103410 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe9c1013e0 0x7efe9c195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.920+0000 7efea3e0a700 1 --2- 172.21.15.69:0/872103410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe9c101cf0 0x7efe9c195eb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:32.956 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.920+0000 7efea3e0a700 1 -- 172.21.15.69:0/872103410 >> 172.21.15.69:0/872103410 conn(0x7efe9c076160 msgr2=0x7efe9c102100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:32.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.920+0000 7efea3e0a700 1 -- 172.21.15.69:0/872103410 shutdown_connections 2023-07-12T15:28:32.957 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:32.920+0000 7efea3e0a700 1 -- 172.21.15.69:0/872103410 wait complete. 2023-07-12T15:28:32.957 INFO:teuthology.orchestra.run.smithi069.stdout:197568495635 2023-07-12T15:28:33.446 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/869618262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-12T15:28:33.446 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3237100334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-12T15:28:33.447 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/872103410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-12T15:28:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.345+0000 7f0b2ba93700 1 -- 172.21.15.69:0/2717304809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b241013b0 msgr2=0x7f0b24101780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.345+0000 7f0b2ba93700 1 --2- 172.21.15.69:0/2717304809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b241013b0 0x7f0b24101780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0b14009990 tx=0x7f0b14009ca0 comp rx=0 tx=0).stop 2023-07-12T15:28:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.345+0000 7f0b2ba93700 1 -- 172.21.15.69:0/2717304809 shutdown_connections 2023-07-12T15:28:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.345+0000 7f0b2ba93700 1 --2- 172.21.15.69:0/2717304809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b2410a790 0x7f0b2410cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.345+0000 7f0b2ba93700 1 --2- 172.21.15.69:0/2717304809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b241013b0 0x7f0b24101780 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.345+0000 7f0b2ba93700 1 --2- 172.21.15.69:0/2717304809 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b24101cc0 0x7f0b2410a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:33.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.345+0000 7f0b2ba93700 1 -- 172.21.15.69:0/2717304809 >> 172.21.15.69:0/2717304809 conn(0x7f0b240761e0 msgr2=0x7f0b240765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:33.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.345+0000 7f0b2ba93700 1 -- 172.21.15.69:0/2717304809 shutdown_connections 2023-07-12T15:28:33.495 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.345+0000 7f0b2ba93700 1 -- 172.21.15.69:0/2717304809 wait complete. 2023-07-12T15:28:33.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2ba93700 1 Processor -- start 2023-07-12T15:28:33.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2ba93700 1 -- start start 2023-07-12T15:28:33.496 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2ba93700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b241013b0 0x7f0b2419b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:33.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2ba93700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b24101cc0 0x7f0b2419bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:33.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2ba93700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b2410a790 0x7f0b24195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:33.497 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2ba93700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b24195eb0 con 0x7f0b24101cc0 2023-07-12T15:28:33.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2ba93700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b24196020 con 0x7f0b2410a790 2023-07-12T15:28:33.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2ba93700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b241961e0 con 0x7f0b241013b0 2023-07-12T15:28:33.498 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2982f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b241013b0 0x7f0b2419b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2902e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b24101cc0 0x7f0b2419bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2982f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b241013b0 0x7f0b2419b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34914/0 (socket says 172.21.15.69:34914) 2023-07-12T15:28:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2982f700 1 -- 172.21.15.69:0/1942453273 learned_addr learned my addr 172.21.15.69:0/1942453273 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2a030700 1 --2- 172.21.15.69:0/1942453273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b2410a790 0x7f0b24195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2982f700 1 -- 172.21.15.69:0/1942453273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b2410a790 msgr2=0x7f0b24195840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:33.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2982f700 1 --2- 172.21.15.69:0/1942453273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b2410a790 0x7f0b24195840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:33.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2982f700 1 -- 172.21.15.69:0/1942453273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b24101cc0 msgr2=0x7f0b2419bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:33.500 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2982f700 1 --2- 172.21.15.69:0/1942453273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b24101cc0 0x7f0b2419bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:33.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2982f700 1 -- 172.21.15.69:0/1942453273 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c009c40 con 0x7f0b241013b0 2023-07-12T15:28:33.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2902e700 1 --2- 172.21.15.69:0/1942453273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b24101cc0 0x7f0b2419bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:28:33.501 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2a030700 1 --2- 172.21.15.69:0/1942453273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b2410a790 0x7f0b24195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:28:33.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2982f700 1 --2- 172.21.15.69:0/1942453273 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b241013b0 0x7f0b2419b7c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0b14006010 tx=0x7f0b14004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:33.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b1affd700 1 -- 172.21.15.69:0/1942453273 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1401d070 con 0x7f0b241013b0 2023-07-12T15:28:33.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b2ba93700 1 -- 172.21.15.69:0/1942453273 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b140095f0 con 0x7f0b241013b0 2023-07-12T15:28:33.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.346+0000 7f0b1affd700 1 -- 172.21.15.69:0/1942453273 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f0b1400fbc0 con 0x7f0b241013b0 2023-07-12T15:28:33.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.347+0000 7f0b1affd700 1 -- 172.21.15.69:0/1942453273 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b140173f0 con 0x7f0b241013b0 2023-07-12T15:28:33.502 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.347+0000 7f0b2ba93700 1 -- 172.21.15.69:0/1942453273 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b241a25e0 con 0x7f0b241013b0 2023-07-12T15:28:33.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.347+0000 7f0b2ba93700 1 -- 172.21.15.69:0/1942453273 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b24104ff0 con 0x7f0b241013b0 2023-07-12T15:28:33.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.348+0000 7f0b1affd700 1 -- 172.21.15.69:0/1942453273 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f0b14022470 con 0x7f0b241013b0 2023-07-12T15:28:33.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.348+0000 7f0b1affd700 1 --2- 172.21.15.69:0/1942453273 >> v2:172.21.15.69:6800/1242716914 conn(0x7f0b1006c810 0x7f0b1006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:33.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.348+0000 7f0b2902e700 1 --2- 172.21.15.69:0/1942453273 >> v2:172.21.15.69:6800/1242716914 conn(0x7f0b1006c810 0x7f0b1006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:33.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.348+0000 7f0b1affd700 1 -- 172.21.15.69:0/1942453273 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f0b1408d110 con 0x7f0b241013b0 2023-07-12T15:28:33.503 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.349+0000 7f0b2902e700 1 --2- 172.21.15.69:0/1942453273 >> v2:172.21.15.69:6800/1242716914 conn(0x7f0b1006c810 0x7f0b1006ecc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0b2410cec0 tx=0x7f0b20009190 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:33.504 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.353+0000 7f0b1affd700 1 -- 172.21.15.69:0/1942453273 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0b1405b030 con 0x7f0b241013b0 2023-07-12T15:28:33.531 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411362 got 77309411364 for osd.2 2023-07-12T15:28:33.532 DEBUG:teuthology.parallel:result is None 2023-07-12T15:28:33.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.553+0000 7f0b2ba93700 1 -- 172.21.15.69:0/1942453273 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f0b24196d50 con 0x7f0b241013b0 2023-07-12T15:28:33.556 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.553+0000 7f0b1affd700 1 -- 172.21.15.69:0/1942453273 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0b1405abc0 con 0x7f0b241013b0 2023-07-12T15:28:33.557 INFO:teuthology.orchestra.run.smithi069.stdout:107374182432 2023-07-12T15:28:33.558 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.555+0000 7f0b2ba93700 1 -- 172.21.15.69:0/1942453273 >> v2:172.21.15.69:6800/1242716914 conn(0x7f0b1006c810 msgr2=0x7f0b1006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:33.558 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.555+0000 7f0b2ba93700 1 --2- 172.21.15.69:0/1942453273 >> v2:172.21.15.69:6800/1242716914 conn(0x7f0b1006c810 0x7f0b1006ecc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0b2410cec0 tx=0x7f0b20009190 comp rx=0 tx=0).stop 2023-07-12T15:28:33.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.555+0000 7f0b2ba93700 1 -- 172.21.15.69:0/1942453273 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b241013b0 msgr2=0x7f0b2419b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:33.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.556+0000 7f0b2ba93700 1 --2- 172.21.15.69:0/1942453273 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b241013b0 0x7f0b2419b7c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0b14006010 tx=0x7f0b14004730 comp rx=0 tx=0).stop 2023-07-12T15:28:33.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.556+0000 7f0b2ba93700 1 -- 172.21.15.69:0/1942453273 shutdown_connections 2023-07-12T15:28:33.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.556+0000 7f0b2ba93700 1 --2- 172.21.15.69:0/1942453273 >> v2:172.21.15.69:6800/1242716914 conn(0x7f0b1006c810 0x7f0b1006ecc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:33.559 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.556+0000 7f0b2ba93700 1 --2- 172.21.15.69:0/1942453273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b2410a790 0x7f0b24195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:33.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.556+0000 7f0b2ba93700 1 --2- 172.21.15.69:0/1942453273 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b241013b0 0x7f0b2419b7c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:33.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.556+0000 7f0b2ba93700 1 --2- 172.21.15.69:0/1942453273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b24101cc0 0x7f0b2419bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:33.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.556+0000 7f0b2ba93700 1 -- 172.21.15.69:0/1942453273 >> 172.21.15.69:0/1942453273 conn(0x7f0b240761e0 msgr2=0x7f0b24102130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:33.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.556+0000 7f0b2ba93700 1 -- 172.21.15.69:0/1942453273 shutdown_connections 2023-07-12T15:28:33.560 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:33.556+0000 7f0b2ba93700 1 -- 172.21.15.69:0/1942453273 wait complete. 2023-07-12T15:28:33.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/869618262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-12T15:28:33.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3237100334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-12T15:28:33.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/872103410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-12T15:28:33.845 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738411 got 34359738413 for osd.0 2023-07-12T15:28:33.845 DEBUG:teuthology.parallel:result is None 2023-07-12T15:28:33.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/869618262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-12T15:28:33.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3237100334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-12T15:28:33.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/872103410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-12T15:28:34.148 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757269 got 163208757271 for osd.5 2023-07-12T15:28:34.148 DEBUG:teuthology.parallel:result is None 2023-07-12T15:28:34.459 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182430 got 107374182432 for osd.3 2023-07-12T15:28:34.460 DEBUG:teuthology.parallel:result is None 2023-07-12T15:28:34.673 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:34 smithi069 ceph-mon[102777]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-12T15:28:34.673 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1942453273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-12T15:28:34.673 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:34 smithi069 ceph-mon[94241]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-12T15:28:34.673 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1942453273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-12T15:28:34.726 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495634 got 197568495635 for osd.6 2023-07-12T15:28:34.726 DEBUG:teuthology.parallel:result is None 2023-07-12T15:28:34.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:28:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:28:34.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:34 smithi145 ceph-mon[95801]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-12T15:28:34.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1942453273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-12T15:28:35.239 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.237+0000 7f16ee9f3700 1 -- 172.21.15.69:0/1035042126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16e80fe6d0 msgr2=0x7f16e80feb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:35.240 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.237+0000 7f16ee9f3700 1 --2- 172.21.15.69:0/1035042126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16e80fe6d0 0x7f16e80feb40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f16dc009990 tx=0x7f16dc009ca0 comp rx=0 tx=0).stop 2023-07-12T15:28:35.240 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.237+0000 7f16ee9f3700 1 -- 172.21.15.69:0/1035042126 shutdown_connections 2023-07-12T15:28:35.240 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.237+0000 7f16ee9f3700 1 --2- 172.21.15.69:0/1035042126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16e80ff080 0x7f16e810cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:35.240 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.237+0000 7f16ee9f3700 1 --2- 172.21.15.69:0/1035042126 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16e8105130 0x7f16e8105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:35.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.237+0000 7f16ee9f3700 1 --2- 172.21.15.69:0/1035042126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16e80fe6d0 0x7f16e80feb40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:35.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.237+0000 7f16ee9f3700 1 -- 172.21.15.69:0/1035042126 >> 172.21.15.69:0/1035042126 conn(0x7f16e8075740 msgr2=0x7f16e8075b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:35.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.237+0000 7f16ee9f3700 1 -- 172.21.15.69:0/1035042126 shutdown_connections 2023-07-12T15:28:35.241 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.237+0000 7f16ee9f3700 1 -- 172.21.15.69:0/1035042126 wait complete. 2023-07-12T15:28:35.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.238+0000 7f16ee9f3700 1 Processor -- start 2023-07-12T15:28:35.242 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.238+0000 7f16ee9f3700 1 -- start start 2023-07-12T15:28:35.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.238+0000 7f16ee9f3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16e80fe6d0 0x7f16e819b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:35.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.238+0000 7f16ee9f3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16e80ff080 0x7f16e819bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:35.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.238+0000 7f16ee9f3700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16e8105130 0x7f16e8195690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:35.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.238+0000 7f16ee9f3700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e8195e40 con 0x7f16e80ff080 2023-07-12T15:28:35.243 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.238+0000 7f16ee9f3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e8195fb0 con 0x7f16e80fe6d0 2023-07-12T15:28:35.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.238+0000 7f16ee9f3700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e8196170 con 0x7f16e8105130 2023-07-12T15:28:35.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.238+0000 7f16e77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16e80ff080 0x7f16e819bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:35.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.238+0000 7f16ecf90700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16e8105130 0x7f16e8195690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:35.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.238+0000 7f16e77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16e80ff080 0x7f16e819bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41500/0 (socket says 172.21.15.69:41500) 2023-07-12T15:28:35.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.238+0000 7f16e77fe700 1 -- 172.21.15.69:0/4032190344 learned_addr learned my addr 172.21.15.69:0/4032190344 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:35.244 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.238+0000 7f16e7fff700 1 --2- 172.21.15.69:0/4032190344 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16e80fe6d0 0x7f16e819b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:35.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.239+0000 7f16ecf90700 1 -- 172.21.15.69:0/4032190344 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16e80fe6d0 msgr2=0x7f16e819b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:35.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.239+0000 7f16ecf90700 1 --2- 172.21.15.69:0/4032190344 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16e80fe6d0 0x7f16e819b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:35.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.239+0000 7f16ecf90700 1 -- 172.21.15.69:0/4032190344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16e80ff080 msgr2=0x7f16e819bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:35.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.239+0000 7f16ecf90700 1 --2- 172.21.15.69:0/4032190344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16e80ff080 0x7f16e819bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:35.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.239+0000 7f16ecf90700 1 -- 172.21.15.69:0/4032190344 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc0095f0 con 0x7f16e8105130 2023-07-12T15:28:35.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.239+0000 7f16e7fff700 1 --2- 172.21.15.69:0/4032190344 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16e80fe6d0 0x7f16e819b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:28:35.245 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.239+0000 7f16e77fe700 1 --2- 172.21.15.69:0/4032190344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16e80ff080 0x7f16e819bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:28:35.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.239+0000 7f16ecf90700 1 --2- 172.21.15.69:0/4032190344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16e8105130 0x7f16e8195690 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f16e000efc0 tx=0x7f16e000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:35.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.239+0000 7f16e57fa700 1 -- 172.21.15.69:0/4032190344 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e0018070 con 0x7f16e8105130 2023-07-12T15:28:35.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.239+0000 7f16ee9f3700 1 -- 172.21.15.69:0/4032190344 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e81963f0 con 0x7f16e8105130 2023-07-12T15:28:35.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.239+0000 7f16e57fa700 1 -- 172.21.15.69:0/4032190344 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f16e000f040 con 0x7f16e8105130 2023-07-12T15:28:35.246 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.239+0000 7f16e57fa700 1 -- 172.21.15.69:0/4032190344 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e000cc50 con 0x7f16e8105130 2023-07-12T15:28:35.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.239+0000 7f16ee9f3700 1 -- 172.21.15.69:0/4032190344 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e81a2620 con 0x7f16e8105130 2023-07-12T15:28:35.247 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.240+0000 7f16ee9f3700 1 -- 172.21.15.69:0/4032190344 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e804e800 con 0x7f16e8105130 2023-07-12T15:28:35.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.241+0000 7f16e57fa700 1 -- 172.21.15.69:0/4032190344 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f16e0003a00 con 0x7f16e8105130 2023-07-12T15:28:35.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.241+0000 7f16e57fa700 1 --2- 172.21.15.69:0/4032190344 >> v2:172.21.15.69:6800/1242716914 conn(0x7f16d006c810 0x7f16d006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:35.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.241+0000 7f16e7fff700 1 --2- 172.21.15.69:0/4032190344 >> v2:172.21.15.69:6800/1242716914 conn(0x7f16d006c810 0x7f16d006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:35.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.241+0000 7f16e57fa700 1 -- 172.21.15.69:0/4032190344 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f16e008b750 con 0x7f16e8105130 2023-07-12T15:28:35.248 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.242+0000 7f16e7fff700 1 --2- 172.21.15.69:0/4032190344 >> v2:172.21.15.69:6800/1242716914 conn(0x7f16d006c810 0x7f16d006ecc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f16d8005fd0 tx=0x7f16d8005e30 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:35.249 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.246+0000 7f16e57fa700 1 -- 172.21.15.69:0/4032190344 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f16e0059600 con 0x7f16e8105130 2023-07-12T15:28:35.455 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.453+0000 7f16ee9f3700 1 -- 172.21.15.69:0/4032190344 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f16e8196c60 con 0x7f16e8105130 2023-07-12T15:28:35.456 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.453+0000 7f16e57fa700 1 -- 172.21.15.69:0/4032190344 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f16e0059190 con 0x7f16e8105130 2023-07-12T15:28:35.456 INFO:teuthology.orchestra.run.smithi069.stdout:128849018908 2023-07-12T15:28:35.457 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.455+0000 7f16ee9f3700 1 -- 172.21.15.69:0/4032190344 >> v2:172.21.15.69:6800/1242716914 conn(0x7f16d006c810 msgr2=0x7f16d006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:35.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.455+0000 7f16ee9f3700 1 --2- 172.21.15.69:0/4032190344 >> v2:172.21.15.69:6800/1242716914 conn(0x7f16d006c810 0x7f16d006ecc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f16d8005fd0 tx=0x7f16d8005e30 comp rx=0 tx=0).stop 2023-07-12T15:28:35.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.455+0000 7f16ee9f3700 1 -- 172.21.15.69:0/4032190344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16e8105130 msgr2=0x7f16e8195690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:35.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.455+0000 7f16ee9f3700 1 --2- 172.21.15.69:0/4032190344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16e8105130 0x7f16e8195690 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f16e000efc0 tx=0x7f16e000c3a0 comp rx=0 tx=0).stop 2023-07-12T15:28:35.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.455+0000 7f16ee9f3700 1 -- 172.21.15.69:0/4032190344 shutdown_connections 2023-07-12T15:28:35.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.455+0000 7f16ee9f3700 1 --2- 172.21.15.69:0/4032190344 >> v2:172.21.15.69:6800/1242716914 conn(0x7f16d006c810 0x7f16d006ecc0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:35.458 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.455+0000 7f16ee9f3700 1 --2- 172.21.15.69:0/4032190344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16e8105130 0x7f16e8195690 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:35.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.455+0000 7f16ee9f3700 1 --2- 172.21.15.69:0/4032190344 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16e80fe6d0 0x7f16e819b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:35.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.455+0000 7f16ee9f3700 1 --2- 172.21.15.69:0/4032190344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16e80ff080 0x7f16e819bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:35.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.455+0000 7f16ee9f3700 1 -- 172.21.15.69:0/4032190344 >> 172.21.15.69:0/4032190344 conn(0x7f16e8075740 msgr2=0x7f16e80fc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:35.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.455+0000 7f16ee9f3700 1 -- 172.21.15.69:0/4032190344 shutdown_connections 2023-07-12T15:28:35.459 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:35.455+0000 7f16ee9f3700 1 -- 172.21.15.69:0/4032190344 wait complete. 2023-07-12T15:28:35.621 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:35 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:35.621 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:35 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:35.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:35 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:35.875 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018906 got 128849018908 for osd.4 2023-07-12T15:28:35.875 DEBUG:teuthology.parallel:result is None 2023-07-12T15:28:35.875 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-07-12T15:28:35.876 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph pg dump --format=json 2023-07-12T15:28:36.149 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:35 smithi069 systemd[1]: Starting Ceph alertmanager.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:28:36.449 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:36 smithi069 podman[132398]: 2023-07-12T15:28:36.450 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:28:36] "GET /metrics HTTP/1.1" 200 219516 "" "Prometheus/2.43.0" 2023-07-12T15:28:36.638 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:36.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-mon[102777]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:36.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4032190344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-12T15:28:36.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-mon[94241]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:36.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4032190344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-12T15:28:36.736 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:28:36] ENGINE Bus STOPPING 2023-07-12T15:28:36.737 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:28:36] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2023-07-12T15:28:36.737 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:28:36] ENGINE Bus STOPPED 2023-07-12T15:28:36.737 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:28:36] ENGINE Bus STARTING 2023-07-12T15:28:36.737 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[132429]: ts=2023-07-12T15:28:36.476Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2023-07-12T15:28:36.737 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[132429]: ts=2023-07-12T15:28:36.476Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2023-07-12T15:28:36.738 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[132429]: ts=2023-07-12T15:28:36.477Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.69 port=9094 2023-07-12T15:28:36.738 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[132429]: ts=2023-07-12T15:28:36.478Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-07-12T15:28:36.738 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:36 smithi069 bash[132398]: 64c01a2fda3bb8d544f47d84a8f9eb26560309cce96c804fa99eedb4fc441923 2023-07-12T15:28:36.738 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:36 smithi069 systemd[1]: Started Ceph alertmanager.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:28:36.738 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[132429]: ts=2023-07-12T15:28:36.502Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-12T15:28:36.738 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[132429]: ts=2023-07-12T15:28:36.503Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-12T15:28:36.738 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[132429]: ts=2023-07-12T15:28:36.506Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2023-07-12T15:28:36.738 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[132429]: ts=2023-07-12T15:28:36.506Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2023-07-12T15:28:36.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:36 smithi145 ceph-mon[95801]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:36.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4032190344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-12T15:28:37.045 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:28:36] ENGINE Serving on http://:::9283 2023-07-12T15:28:37.045 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:28:36] ENGINE Bus STARTED 2023-07-12T15:28:37.709 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[94241]: Regenerating cephadm self-signed grafana TLS certificates 2023-07-12T15:28:37.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-12T15:28:37.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-12T15:28:37.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[94241]: Deploying daemon grafana.a on smithi145 2023-07-12T15:28:37.711 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[94241]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:37.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[102777]: Regenerating cephadm self-signed grafana TLS certificates 2023-07-12T15:28:37.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.712 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-12T15:28:37.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-12T15:28:37.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[102777]: Deploying daemon grafana.a on smithi145 2023-07-12T15:28:37.713 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:37 smithi069 ceph-mon[102777]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:37.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:37 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:37 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:37 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:37 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:37 smithi145 ceph-mon[95801]: Regenerating cephadm self-signed grafana TLS certificates 2023-07-12T15:28:37.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:37 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:37 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:37 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-12T15:28:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:37 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-07-12T15:28:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:37 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:37 smithi145 ceph-mon[95801]: Deploying daemon grafana.a on smithi145 2023-07-12T15:28:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:37 smithi145 ceph-mon[95801]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:37.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.883+0000 7f541fda1700 1 -- 172.21.15.69:0/1579463288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5418101610 msgr2=0x7f541810c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:37.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.883+0000 7f541fda1700 1 --2- 172.21.15.69:0/1579463288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5418101610 0x7f541810c000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f541000b0d0 tx=0x7f541000b3e0 comp rx=0 tx=0).stop 2023-07-12T15:28:37.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.883+0000 7f541fda1700 1 -- 172.21.15.69:0/1579463288 shutdown_connections 2023-07-12T15:28:37.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.883+0000 7f541fda1700 1 --2- 172.21.15.69:0/1579463288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5418101610 0x7f541810c000 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:37.886 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.883+0000 7f541fda1700 1 --2- 172.21.15.69:0/1579463288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54180ff400 0x7f54180ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:37.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.883+0000 7f541fda1700 1 --2- 172.21.15.69:0/1579463288 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5418100c60 0x7f54181010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:37.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.883+0000 7f541fda1700 1 -- 172.21.15.69:0/1579463288 >> 172.21.15.69:0/1579463288 conn(0x7f5418076150 msgr2=0x7f5418076550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:37.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.883+0000 7f541fda1700 1 -- 172.21.15.69:0/1579463288 shutdown_connections 2023-07-12T15:28:37.887 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.883+0000 7f541fda1700 1 -- 172.21.15.69:0/1579463288 wait complete. 2023-07-12T15:28:37.888 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.884+0000 7f541fda1700 1 Processor -- start 2023-07-12T15:28:37.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.884+0000 7f541fda1700 1 -- start start 2023-07-12T15:28:37.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.884+0000 7f541fda1700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54180ff400 0x7f541819b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:37.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.884+0000 7f541fda1700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5418100c60 0x7f541819beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:37.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.884+0000 7f541fda1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5418101610 0x7f54181959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:37.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.884+0000 7f541fda1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5418196060 con 0x7f5418101610 2023-07-12T15:28:37.889 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.884+0000 7f541fda1700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54181961d0 con 0x7f5418100c60 2023-07-12T15:28:37.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.884+0000 7f541fda1700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5418196390 con 0x7f54180ff400 2023-07-12T15:28:37.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.884+0000 7f541e33e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5418101610 0x7f54181959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:37.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.884+0000 7f541e33e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5418101610 0x7f54181959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41524/0 (socket says 172.21.15.69:41524) 2023-07-12T15:28:37.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.884+0000 7f541e33e700 1 -- 172.21.15.69:0/146916530 learned_addr learned my addr 172.21.15.69:0/146916530 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:37.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.884+0000 7f541d33c700 1 --2- 172.21.15.69:0/146916530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5418100c60 0x7f541819beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:37.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.884+0000 7f541db3d700 1 --2- 172.21.15.69:0/146916530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54180ff400 0x7f541819b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:37.890 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.885+0000 7f541e33e700 1 -- 172.21.15.69:0/146916530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54180ff400 msgr2=0x7f541819b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:37.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.885+0000 7f541e33e700 1 --2- 172.21.15.69:0/146916530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54180ff400 0x7f541819b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:37.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.885+0000 7f541e33e700 1 -- 172.21.15.69:0/146916530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5418100c60 msgr2=0x7f541819beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:37.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.885+0000 7f541e33e700 1 --2- 172.21.15.69:0/146916530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5418100c60 0x7f541819beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:37.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.885+0000 7f541e33e700 1 -- 172.21.15.69:0/146916530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5408009520 con 0x7f5418101610 2023-07-12T15:28:37.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.885+0000 7f541e33e700 1 --2- 172.21.15.69:0/146916530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5418101610 0x7f54181959f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5410008ed0 tx=0x7f5410008f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:37.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.885+0000 7f540effd700 1 -- 172.21.15.69:0/146916530 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541000e040 con 0x7f5418101610 2023-07-12T15:28:37.891 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.885+0000 7f540effd700 1 -- 172.21.15.69:0/146916530 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f5410003e90 con 0x7f5418101610 2023-07-12T15:28:37.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.885+0000 7f540effd700 1 -- 172.21.15.69:0/146916530 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541001b6c0 con 0x7f5418101610 2023-07-12T15:28:37.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.885+0000 7f541fda1700 1 -- 172.21.15.69:0/146916530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5410009d00 con 0x7f5418101610 2023-07-12T15:28:37.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.885+0000 7f541fda1700 1 -- 172.21.15.69:0/146916530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541810c900 con 0x7f5418101610 2023-07-12T15:28:37.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.886+0000 7f541fda1700 1 -- 172.21.15.69:0/146916530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541804e800 con 0x7f5418101610 2023-07-12T15:28:37.892 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.887+0000 7f540effd700 1 -- 172.21.15.69:0/146916530 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f5410003a00 con 0x7f5418101610 2023-07-12T15:28:37.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.887+0000 7f540effd700 1 --2- 172.21.15.69:0/146916530 >> v2:172.21.15.69:6800/1242716914 conn(0x7f5404070c20 0x7f54040730d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:37.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.887+0000 7f540effd700 1 -- 172.21.15.69:0/146916530 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f541008cb80 con 0x7f5418101610 2023-07-12T15:28:37.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.887+0000 7f541db3d700 1 --2- 172.21.15.69:0/146916530 >> v2:172.21.15.69:6800/1242716914 conn(0x7f5404070c20 0x7f54040730d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:37.893 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.887+0000 7f541db3d700 1 --2- 172.21.15.69:0/146916530 >> v2:172.21.15.69:6800/1242716914 conn(0x7f5404070c20 0x7f54040730d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5408009df0 tx=0x7f5408009260 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:37.894 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:37.891+0000 7f540effd700 1 -- 172.21.15.69:0/146916530 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f541005aa30 con 0x7f5418101610 2023-07-12T15:28:38.077 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.074+0000 7f541fda1700 1 -- 172.21.15.69:0/146916530 --> v2:172.21.15.69:6800/1242716914 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5418196f00 con 0x7f5404070c20 2023-07-12T15:28:38.087 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.085+0000 7f540effd700 1 -- 172.21.15.69:0/146916530 <== mgr.24524 v2:172.21.15.69:6800/1242716914 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+384258 (secure 0 0 0) 0x7f5418196f00 con 0x7f5404070c20 2023-07-12T15:28:38.091 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:28:38.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.089+0000 7f541fda1700 1 -- 172.21.15.69:0/146916530 >> v2:172.21.15.69:6800/1242716914 conn(0x7f5404070c20 msgr2=0x7f54040730d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:38.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.089+0000 7f541fda1700 1 --2- 172.21.15.69:0/146916530 >> v2:172.21.15.69:6800/1242716914 conn(0x7f5404070c20 0x7f54040730d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5408009df0 tx=0x7f5408009260 comp rx=0 tx=0).stop 2023-07-12T15:28:38.093 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.089+0000 7f541fda1700 1 -- 172.21.15.69:0/146916530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5418101610 msgr2=0x7f54181959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:38.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.089+0000 7f541fda1700 1 --2- 172.21.15.69:0/146916530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5418101610 0x7f54181959f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5410008ed0 tx=0x7f5410008f00 comp rx=0 tx=0).stop 2023-07-12T15:28:38.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.089+0000 7f541fda1700 1 -- 172.21.15.69:0/146916530 shutdown_connections 2023-07-12T15:28:38.094 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.089+0000 7f541fda1700 1 --2- 172.21.15.69:0/146916530 >> v2:172.21.15.69:6800/1242716914 conn(0x7f5404070c20 0x7f54040730d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:38.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.089+0000 7f541fda1700 1 --2- 172.21.15.69:0/146916530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5418101610 0x7f54181959f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:38.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.089+0000 7f541fda1700 1 --2- 172.21.15.69:0/146916530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54180ff400 0x7f541819b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:38.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.089+0000 7f541fda1700 1 --2- 172.21.15.69:0/146916530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5418100c60 0x7f541819beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:38.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.089+0000 7f541fda1700 1 -- 172.21.15.69:0/146916530 >> 172.21.15.69:0/146916530 conn(0x7f5418076150 msgr2=0x7f54180fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:38.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.089+0000 7f541fda1700 1 -- 172.21.15.69:0/146916530 shutdown_connections 2023-07-12T15:28:38.095 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:38.089+0000 7f541fda1700 1 -- 172.21.15.69:0/146916530 wait complete. 2023-07-12T15:28:38.095 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-07-12T15:28:38.588 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-07-12T15:28:37.372180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596224,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":981,"num_read_kb":827,"num_write":609,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":622,"ondisk_log_size":622,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":222668,"kb_used_data":7052,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":749738548,"statfs":{"total":767960285184,"available":767732273152,"internally_reserved":0,"allocated":7221248,"data_stored":3883642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002708"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166198+0000","last_change":"2023-07-12T15:27:46.719957+0000","last_active":"2023-07-12T15:27:50.166198+0000","last_peered":"2023-07-12T15:27:50.166198+0000","last_clean":"2023-07-12T15:27:50.166198+0000","last_became_active":"2023-07-12T15:27:46.719622+0000","last_became_peered":"2023-07-12T15:27:46.719622+0000","last_unstale":"2023-07-12T15:27:50.166198+0000","last_undegraded":"2023-07-12T15:27:50.166198+0000","last_fullsized":"2023-07-12T15:27:50.166198+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:34:02.770908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166029+0000","last_change":"2023-07-12T15:27:44.145601+0000","last_active":"2023-07-12T15:27:50.166029+0000","last_peered":"2023-07-12T15:27:50.166029+0000","last_clean":"2023-07-12T15:27:50.166029+0000","last_became_active":"2023-07-12T15:27:44.145284+0000","last_became_peered":"2023-07-12T15:27:44.145284+0000","last_unstale":"2023-07-12T15:27:50.166029+0000","last_undegraded":"2023-07-12T15:27:50.166029+0000","last_fullsized":"2023-07-12T15:27:50.166029+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:01:19.271063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166333+0000","last_change":"2023-07-12T15:27:40.130960+0000","last_active":"2023-07-12T15:27:50.166333+0000","last_peered":"2023-07-12T15:27:50.166333+0000","last_clean":"2023-07-12T15:27:50.166333+0000","last_became_active":"2023-07-12T15:27:40.130721+0000","last_became_peered":"2023-07-12T15:27:40.130721+0000","last_unstale":"2023-07-12T15:27:50.166333+0000","last_undegraded":"2023-07-12T15:27:50.166333+0000","last_fullsized":"2023-07-12T15:27:50.166333+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:22:03.109709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165359+0000","last_change":"2023-07-12T15:27:42.134026+0000","last_active":"2023-07-12T15:27:50.165359+0000","last_peered":"2023-07-12T15:27:50.165359+0000","last_clean":"2023-07-12T15:27:50.165359+0000","last_became_active":"2023-07-12T15:27:42.133731+0000","last_became_peered":"2023-07-12T15:27:42.133731+0000","last_unstale":"2023-07-12T15:27:50.165359+0000","last_undegraded":"2023-07-12T15:27:50.165359+0000","last_fullsized":"2023-07-12T15:27:50.165359+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:23:38.384520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166460+0000","last_change":"2023-07-12T15:27:46.163608+0000","last_active":"2023-07-12T15:27:50.166460+0000","last_peered":"2023-07-12T15:27:50.166460+0000","last_clean":"2023-07-12T15:27:50.166460+0000","last_became_active":"2023-07-12T15:27:46.163185+0000","last_became_peered":"2023-07-12T15:27:46.163185+0000","last_unstale":"2023-07-12T15:27:50.166460+0000","last_undegraded":"2023-07-12T15:27:50.166460+0000","last_fullsized":"2023-07-12T15:27:50.166460+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:51:17.234063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954488+0000","last_change":"2023-07-12T15:27:44.148220+0000","last_active":"2023-07-12T15:27:49.954488+0000","last_peered":"2023-07-12T15:27:49.954488+0000","last_clean":"2023-07-12T15:27:49.954488+0000","last_became_active":"2023-07-12T15:27:44.147914+0000","last_became_peered":"2023-07-12T15:27:44.147914+0000","last_unstale":"2023-07-12T15:27:49.954488+0000","last_undegraded":"2023-07-12T15:27:49.954488+0000","last_fullsized":"2023-07-12T15:27:49.954488+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:33:22.313311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"60'1","reported_seq":25,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166799+0000","last_change":"2023-07-12T15:27:40.129257+0000","last_active":"2023-07-12T15:27:50.166799+0000","last_peered":"2023-07-12T15:27:50.166799+0000","last_clean":"2023-07-12T15:27:50.166799+0000","last_became_active":"2023-07-12T15:27:40.128960+0000","last_became_peered":"2023-07-12T15:27:40.128960+0000","last_unstale":"2023-07-12T15:27:50.166799+0000","last_undegraded":"2023-07-12T15:27:50.166799+0000","last_fullsized":"2023-07-12T15:27:50.166799+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:36:36.864844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165931+0000","last_change":"2023-07-12T15:27:42.135394+0000","last_active":"2023-07-12T15:27:50.165931+0000","last_peered":"2023-07-12T15:27:50.165931+0000","last_clean":"2023-07-12T15:27:50.165931+0000","last_became_active":"2023-07-12T15:27:42.134943+0000","last_became_peered":"2023-07-12T15:27:42.134943+0000","last_unstale":"2023-07-12T15:27:50.165931+0000","last_undegraded":"2023-07-12T15:27:50.165931+0000","last_fullsized":"2023-07-12T15:27:50.165931+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:14:16.773828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952302+0000","last_change":"2023-07-12T15:27:46.164091+0000","last_active":"2023-07-12T15:27:49.952302+0000","last_peered":"2023-07-12T15:27:49.952302+0000","last_clean":"2023-07-12T15:27:49.952302+0000","last_became_active":"2023-07-12T15:27:46.163796+0000","last_became_peered":"2023-07-12T15:27:46.163796+0000","last_unstale":"2023-07-12T15:27:49.952302+0000","last_undegraded":"2023-07-12T15:27:49.952302+0000","last_fullsized":"2023-07-12T15:27:49.952302+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T20:54:16.307133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166124+0000","last_change":"2023-07-12T15:27:44.155072+0000","last_active":"2023-07-12T15:27:50.166124+0000","last_peered":"2023-07-12T15:27:50.166124+0000","last_clean":"2023-07-12T15:27:50.166124+0000","last_became_active":"2023-07-12T15:27:44.154743+0000","last_became_peered":"2023-07-12T15:27:44.154743+0000","last_unstale":"2023-07-12T15:27:50.166124+0000","last_undegraded":"2023-07-12T15:27:50.166124+0000","last_fullsized":"2023-07-12T15:27:50.166124+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:52:24.294886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167786+0000","last_change":"2023-07-12T15:27:40.134604+0000","last_active":"2023-07-12T15:27:50.167786+0000","last_peered":"2023-07-12T15:27:50.167786+0000","last_clean":"2023-07-12T15:27:50.167786+0000","last_became_active":"2023-07-12T15:27:40.134306+0000","last_became_peered":"2023-07-12T15:27:40.134306+0000","last_unstale":"2023-07-12T15:27:50.167786+0000","last_undegraded":"2023-07-12T15:27:50.167786+0000","last_fullsized":"2023-07-12T15:27:50.167786+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:24:48.235457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":27,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166635+0000","last_change":"2023-07-12T15:27:42.138719+0000","last_active":"2023-07-12T15:27:50.166635+0000","last_peered":"2023-07-12T15:27:50.166635+0000","last_clean":"2023-07-12T15:27:50.166635+0000","last_became_active":"2023-07-12T15:27:42.138441+0000","last_became_peered":"2023-07-12T15:27:42.138441+0000","last_unstale":"2023-07-12T15:27:50.166635+0000","last_undegraded":"2023-07-12T15:27:50.166635+0000","last_fullsized":"2023-07-12T15:27:50.166635+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:40:34.302700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"67'1","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166957+0000","last_change":"2023-07-12T15:27:46.169583+0000","last_active":"2023-07-12T15:27:50.166957+0000","last_peered":"2023-07-12T15:27:50.166957+0000","last_clean":"2023-07-12T15:27:50.166957+0000","last_became_active":"2023-07-12T15:27:46.169216+0000","last_became_peered":"2023-07-12T15:27:46.169216+0000","last_unstale":"2023-07-12T15:27:50.166957+0000","last_undegraded":"2023-07-12T15:27:50.166957+0000","last_fullsized":"2023-07-12T15:27:50.166957+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:11:59.801697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952583+0000","last_change":"2023-07-12T15:27:44.156992+0000","last_active":"2023-07-12T15:27:49.952583+0000","last_peered":"2023-07-12T15:27:49.952583+0000","last_clean":"2023-07-12T15:27:49.952583+0000","last_became_active":"2023-07-12T15:27:44.156668+0000","last_became_peered":"2023-07-12T15:27:44.156668+0000","last_unstale":"2023-07-12T15:27:49.952583+0000","last_undegraded":"2023-07-12T15:27:49.952583+0000","last_fullsized":"2023-07-12T15:27:49.952583+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:41:39.142746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165923+0000","last_change":"2023-07-12T15:27:40.133664+0000","last_active":"2023-07-12T15:27:50.165923+0000","last_peered":"2023-07-12T15:27:50.165923+0000","last_clean":"2023-07-12T15:27:50.165923+0000","last_became_active":"2023-07-12T15:27:40.131966+0000","last_became_peered":"2023-07-12T15:27:40.131966+0000","last_unstale":"2023-07-12T15:27:50.165923+0000","last_undegraded":"2023-07-12T15:27:50.165923+0000","last_fullsized":"2023-07-12T15:27:50.165923+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:37:27.700710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166539+0000","last_change":"2023-07-12T15:27:42.135430+0000","last_active":"2023-07-12T15:27:50.166539+0000","last_peered":"2023-07-12T15:27:50.166539+0000","last_clean":"2023-07-12T15:27:50.166539+0000","last_became_active":"2023-07-12T15:27:42.135174+0000","last_became_peered":"2023-07-12T15:27:42.135174+0000","last_unstale":"2023-07-12T15:27:50.166539+0000","last_undegraded":"2023-07-12T15:27:50.166539+0000","last_fullsized":"2023-07-12T15:27:50.166539+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:22:43.754172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166903+0000","last_change":"2023-07-12T15:27:46.717310+0000","last_active":"2023-07-12T15:27:50.166903+0000","last_peered":"2023-07-12T15:27:50.166903+0000","last_clean":"2023-07-12T15:27:50.166903+0000","last_became_active":"2023-07-12T15:27:46.716925+0000","last_became_peered":"2023-07-12T15:27:46.716925+0000","last_unstale":"2023-07-12T15:27:50.166903+0000","last_undegraded":"2023-07-12T15:27:50.166903+0000","last_fullsized":"2023-07-12T15:27:50.166903+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:33:10.467570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167066+0000","last_change":"2023-07-12T15:27:44.149841+0000","last_active":"2023-07-12T15:27:50.167066+0000","last_peered":"2023-07-12T15:27:50.167066+0000","last_clean":"2023-07-12T15:27:50.167066+0000","last_became_active":"2023-07-12T15:27:44.149447+0000","last_became_peered":"2023-07-12T15:27:44.149447+0000","last_unstale":"2023-07-12T15:27:50.167066+0000","last_undegraded":"2023-07-12T15:27:50.167066+0000","last_fullsized":"2023-07-12T15:27:50.167066+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:21:42.788690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167404+0000","last_change":"2023-07-12T15:27:40.130768+0000","last_active":"2023-07-12T15:27:50.167404+0000","last_peered":"2023-07-12T15:27:50.167404+0000","last_clean":"2023-07-12T15:27:50.167404+0000","last_became_active":"2023-07-12T15:27:40.130515+0000","last_became_peered":"2023-07-12T15:27:40.130515+0000","last_unstale":"2023-07-12T15:27:50.167404+0000","last_undegraded":"2023-07-12T15:27:50.167404+0000","last_fullsized":"2023-07-12T15:27:50.167404+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T15:29:43.002441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":40,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165470+0000","last_change":"2023-07-12T15:27:42.137841+0000","last_active":"2023-07-12T15:27:50.165470+0000","last_peered":"2023-07-12T15:27:50.165470+0000","last_clean":"2023-07-12T15:27:50.165470+0000","last_became_active":"2023-07-12T15:27:42.137587+0000","last_became_peered":"2023-07-12T15:27:42.137587+0000","last_unstale":"2023-07-12T15:27:50.165470+0000","last_undegraded":"2023-07-12T15:27:50.165470+0000","last_fullsized":"2023-07-12T15:27:50.165470+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:40:54.563480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167000+0000","last_change":"2023-07-12T15:27:46.717815+0000","last_active":"2023-07-12T15:27:50.167000+0000","last_peered":"2023-07-12T15:27:50.167000+0000","last_clean":"2023-07-12T15:27:50.167000+0000","last_became_active":"2023-07-12T15:27:46.717538+0000","last_became_peered":"2023-07-12T15:27:46.717538+0000","last_unstale":"2023-07-12T15:27:50.167000+0000","last_undegraded":"2023-07-12T15:27:50.167000+0000","last_fullsized":"2023-07-12T15:27:50.167000+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:31:05.594988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956314+0000","last_change":"2023-07-12T15:27:44.145926+0000","last_active":"2023-07-12T15:27:49.956314+0000","last_peered":"2023-07-12T15:27:49.956314+0000","last_clean":"2023-07-12T15:27:49.956314+0000","last_became_active":"2023-07-12T15:27:44.145632+0000","last_became_peered":"2023-07-12T15:27:44.145632+0000","last_unstale":"2023-07-12T15:27:49.956314+0000","last_undegraded":"2023-07-12T15:27:49.956314+0000","last_fullsized":"2023-07-12T15:27:49.956314+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:55:14.743990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166632+0000","last_change":"2023-07-12T15:27:40.133269+0000","last_active":"2023-07-12T15:27:50.166632+0000","last_peered":"2023-07-12T15:27:50.166632+0000","last_clean":"2023-07-12T15:27:50.166632+0000","last_became_active":"2023-07-12T15:27:40.132972+0000","last_became_peered":"2023-07-12T15:27:40.132972+0000","last_unstale":"2023-07-12T15:27:50.166632+0000","last_undegraded":"2023-07-12T15:27:50.166632+0000","last_fullsized":"2023-07-12T15:27:50.166632+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:40:47.394603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956678+0000","last_change":"2023-07-12T15:27:42.133950+0000","last_active":"2023-07-12T15:27:49.956678+0000","last_peered":"2023-07-12T15:27:49.956678+0000","last_clean":"2023-07-12T15:27:49.956678+0000","last_became_active":"2023-07-12T15:27:42.133660+0000","last_became_peered":"2023-07-12T15:27:42.133660+0000","last_unstale":"2023-07-12T15:27:49.956678+0000","last_undegraded":"2023-07-12T15:27:49.956678+0000","last_fullsized":"2023-07-12T15:27:49.956678+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:20:03.074678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955396+0000","last_change":"2023-07-12T15:27:46.160478+0000","last_active":"2023-07-12T15:27:49.955396+0000","last_peered":"2023-07-12T15:27:49.955396+0000","last_clean":"2023-07-12T15:27:49.955396+0000","last_became_active":"2023-07-12T15:27:46.160198+0000","last_became_peered":"2023-07-12T15:27:46.160198+0000","last_unstale":"2023-07-12T15:27:49.955396+0000","last_undegraded":"2023-07-12T15:27:49.955396+0000","last_fullsized":"2023-07-12T15:27:49.955396+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T20:48:27.514758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166821+0000","last_change":"2023-07-12T15:27:44.150054+0000","last_active":"2023-07-12T15:27:50.166821+0000","last_peered":"2023-07-12T15:27:50.166821+0000","last_clean":"2023-07-12T15:27:50.166821+0000","last_became_active":"2023-07-12T15:27:44.149751+0000","last_became_peered":"2023-07-12T15:27:44.149751+0000","last_unstale":"2023-07-12T15:27:50.166821+0000","last_undegraded":"2023-07-12T15:27:50.166821+0000","last_fullsized":"2023-07-12T15:27:50.166821+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:58:42.614488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167064+0000","last_change":"2023-07-12T15:27:40.128889+0000","last_active":"2023-07-12T15:27:50.167064+0000","last_peered":"2023-07-12T15:27:50.167064+0000","last_clean":"2023-07-12T15:27:50.167064+0000","last_became_active":"2023-07-12T15:27:40.128651+0000","last_became_peered":"2023-07-12T15:27:40.128651+0000","last_unstale":"2023-07-12T15:27:50.167064+0000","last_undegraded":"2023-07-12T15:27:50.167064+0000","last_fullsized":"2023-07-12T15:27:50.167064+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:15:47.395585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166871+0000","last_change":"2023-07-12T15:27:42.131973+0000","last_active":"2023-07-12T15:27:50.166871+0000","last_peered":"2023-07-12T15:27:50.166871+0000","last_clean":"2023-07-12T15:27:50.166871+0000","last_became_active":"2023-07-12T15:27:42.131669+0000","last_became_peered":"2023-07-12T15:27:42.131669+0000","last_unstale":"2023-07-12T15:27:50.166871+0000","last_undegraded":"2023-07-12T15:27:50.166871+0000","last_fullsized":"2023-07-12T15:27:50.166871+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:46:42.718907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166252+0000","last_change":"2023-07-12T15:27:46.167326+0000","last_active":"2023-07-12T15:27:50.166252+0000","last_peered":"2023-07-12T15:27:50.166252+0000","last_clean":"2023-07-12T15:27:50.166252+0000","last_became_active":"2023-07-12T15:27:46.167052+0000","last_became_peered":"2023-07-12T15:27:46.167052+0000","last_unstale":"2023-07-12T15:27:50.166252+0000","last_undegraded":"2023-07-12T15:27:50.166252+0000","last_fullsized":"2023-07-12T15:27:50.166252+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:10:15.773772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'11","reported_seq":43,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:28.814440+0000","last_change":"2023-07-12T15:27:44.152180+0000","last_active":"2023-07-12T15:28:28.814440+0000","last_peered":"2023-07-12T15:28:28.814440+0000","last_clean":"2023-07-12T15:28:28.814440+0000","last_became_active":"2023-07-12T15:27:44.151874+0000","last_became_peered":"2023-07-12T15:27:44.151874+0000","last_unstale":"2023-07-12T15:28:28.814440+0000","last_undegraded":"2023-07-12T15:28:28.814440+0000","last_fullsized":"2023-07-12T15:28:28.814440+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:58:02.492468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952222+0000","last_change":"2023-07-12T15:27:40.134571+0000","last_active":"2023-07-12T15:27:49.952222+0000","last_peered":"2023-07-12T15:27:49.952222+0000","last_clean":"2023-07-12T15:27:49.952222+0000","last_became_active":"2023-07-12T15:27:40.134273+0000","last_became_peered":"2023-07-12T15:27:40.134273+0000","last_unstale":"2023-07-12T15:27:49.952222+0000","last_undegraded":"2023-07-12T15:27:49.952222+0000","last_fullsized":"2023-07-12T15:27:49.952222+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:36:11.998289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166739+0000","last_change":"2023-07-12T15:27:42.136459+0000","last_active":"2023-07-12T15:27:50.166739+0000","last_peered":"2023-07-12T15:27:50.166739+0000","last_clean":"2023-07-12T15:27:50.166739+0000","last_became_active":"2023-07-12T15:27:42.136026+0000","last_became_peered":"2023-07-12T15:27:42.136026+0000","last_unstale":"2023-07-12T15:27:50.166739+0000","last_undegraded":"2023-07-12T15:27:50.166739+0000","last_fullsized":"2023-07-12T15:27:50.166739+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:03:36.934879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167358+0000","last_change":"2023-07-12T15:27:46.717953+0000","last_active":"2023-07-12T15:27:50.167358+0000","last_peered":"2023-07-12T15:27:50.167358+0000","last_clean":"2023-07-12T15:27:50.167358+0000","last_became_active":"2023-07-12T15:27:46.717380+0000","last_became_peered":"2023-07-12T15:27:46.717380+0000","last_unstale":"2023-07-12T15:27:50.167358+0000","last_undegraded":"2023-07-12T15:27:50.167358+0000","last_fullsized":"2023-07-12T15:27:50.167358+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:19:21.542061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.168343+0000","last_change":"2023-07-12T15:27:44.155961+0000","last_active":"2023-07-12T15:27:50.168343+0000","last_peered":"2023-07-12T15:27:50.168343+0000","last_clean":"2023-07-12T15:27:50.168343+0000","last_became_active":"2023-07-12T15:27:44.155640+0000","last_became_peered":"2023-07-12T15:27:44.155640+0000","last_unstale":"2023-07-12T15:27:50.168343+0000","last_undegraded":"2023-07-12T15:27:50.168343+0000","last_fullsized":"2023-07-12T15:27:50.168343+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:18:07.314044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167189+0000","last_change":"2023-07-12T15:27:40.130730+0000","last_active":"2023-07-12T15:27:50.167189+0000","last_peered":"2023-07-12T15:27:50.167189+0000","last_clean":"2023-07-12T15:27:50.167189+0000","last_became_active":"2023-07-12T15:27:40.130412+0000","last_became_peered":"2023-07-12T15:27:40.130412+0000","last_unstale":"2023-07-12T15:27:50.167189+0000","last_undegraded":"2023-07-12T15:27:50.167189+0000","last_fullsized":"2023-07-12T15:27:50.167189+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:36:04.948411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167413+0000","last_change":"2023-07-12T15:27:42.132862+0000","last_active":"2023-07-12T15:27:50.167413+0000","last_peered":"2023-07-12T15:27:50.167413+0000","last_clean":"2023-07-12T15:27:50.167413+0000","last_became_active":"2023-07-12T15:27:42.132592+0000","last_became_peered":"2023-07-12T15:27:42.132592+0000","last_unstale":"2023-07-12T15:27:50.167413+0000","last_undegraded":"2023-07-12T15:27:50.167413+0000","last_fullsized":"2023-07-12T15:27:50.167413+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:36:19.909414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165953+0000","last_change":"2023-07-12T15:27:46.167372+0000","last_active":"2023-07-12T15:27:50.165953+0000","last_peered":"2023-07-12T15:27:50.165953+0000","last_clean":"2023-07-12T15:27:50.165953+0000","last_became_active":"2023-07-12T15:27:46.167024+0000","last_became_peered":"2023-07-12T15:27:46.167024+0000","last_unstale":"2023-07-12T15:27:50.165953+0000","last_undegraded":"2023-07-12T15:27:50.165953+0000","last_fullsized":"2023-07-12T15:27:50.165953+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:40:36.392499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167636+0000","last_change":"2023-07-12T15:27:44.158812+0000","last_active":"2023-07-12T15:27:50.167636+0000","last_peered":"2023-07-12T15:27:50.167636+0000","last_clean":"2023-07-12T15:27:50.167636+0000","last_became_active":"2023-07-12T15:27:44.158484+0000","last_became_peered":"2023-07-12T15:27:44.158484+0000","last_unstale":"2023-07-12T15:27:50.167636+0000","last_undegraded":"2023-07-12T15:27:50.167636+0000","last_fullsized":"2023-07-12T15:27:50.167636+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T20:30:56.007120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952742+0000","last_change":"2023-07-12T15:27:40.133310+0000","last_active":"2023-07-12T15:27:49.952742+0000","last_peered":"2023-07-12T15:27:49.952742+0000","last_clean":"2023-07-12T15:27:49.952742+0000","last_became_active":"2023-07-12T15:27:40.132983+0000","last_became_peered":"2023-07-12T15:27:40.132983+0000","last_unstale":"2023-07-12T15:27:49.952742+0000","last_undegraded":"2023-07-12T15:27:49.952742+0000","last_fullsized":"2023-07-12T15:27:49.952742+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:19:34.869750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":23,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165332+0000","last_change":"2023-07-12T15:27:42.136712+0000","last_active":"2023-07-12T15:27:50.165332+0000","last_peered":"2023-07-12T15:27:50.165332+0000","last_clean":"2023-07-12T15:27:50.165332+0000","last_became_active":"2023-07-12T15:27:42.136414+0000","last_became_peered":"2023-07-12T15:27:42.136414+0000","last_unstale":"2023-07-12T15:27:50.165332+0000","last_undegraded":"2023-07-12T15:27:50.165332+0000","last_fullsized":"2023-07-12T15:27:50.165332+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:35:44.585898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166813+0000","last_change":"2023-07-12T15:27:46.167443+0000","last_active":"2023-07-12T15:27:50.166813+0000","last_peered":"2023-07-12T15:27:50.166813+0000","last_clean":"2023-07-12T15:27:50.166813+0000","last_became_active":"2023-07-12T15:27:46.167105+0000","last_became_peered":"2023-07-12T15:27:46.167105+0000","last_unstale":"2023-07-12T15:27:50.166813+0000","last_undegraded":"2023-07-12T15:27:50.166813+0000","last_fullsized":"2023-07-12T15:27:50.166813+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:29:07.739097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954765+0000","last_change":"2023-07-12T15:27:44.154978+0000","last_active":"2023-07-12T15:27:49.954765+0000","last_peered":"2023-07-12T15:27:49.954765+0000","last_clean":"2023-07-12T15:27:49.954765+0000","last_became_active":"2023-07-12T15:27:44.154696+0000","last_became_peered":"2023-07-12T15:27:44.154696+0000","last_unstale":"2023-07-12T15:27:49.954765+0000","last_undegraded":"2023-07-12T15:27:49.954765+0000","last_fullsized":"2023-07-12T15:27:49.954765+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:18:44.337926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956025+0000","last_change":"2023-07-12T15:27:40.128722+0000","last_active":"2023-07-12T15:27:49.956025+0000","last_peered":"2023-07-12T15:27:49.956025+0000","last_clean":"2023-07-12T15:27:49.956025+0000","last_became_active":"2023-07-12T15:27:40.128430+0000","last_became_peered":"2023-07-12T15:27:40.128430+0000","last_unstale":"2023-07-12T15:27:49.956025+0000","last_undegraded":"2023-07-12T15:27:49.956025+0000","last_fullsized":"2023-07-12T15:27:49.956025+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:25:48.743883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166575+0000","last_change":"2023-07-12T15:27:42.135381+0000","last_active":"2023-07-12T15:27:50.166575+0000","last_peered":"2023-07-12T15:27:50.166575+0000","last_clean":"2023-07-12T15:27:50.166575+0000","last_became_active":"2023-07-12T15:27:42.135032+0000","last_became_peered":"2023-07-12T15:27:42.135032+0000","last_unstale":"2023-07-12T15:27:50.166575+0000","last_undegraded":"2023-07-12T15:27:50.166575+0000","last_fullsized":"2023-07-12T15:27:50.166575+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:09:24.118737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166336+0000","last_change":"2023-07-12T15:27:46.169664+0000","last_active":"2023-07-12T15:27:50.166336+0000","last_peered":"2023-07-12T15:27:50.166336+0000","last_clean":"2023-07-12T15:27:50.166336+0000","last_became_active":"2023-07-12T15:27:46.169282+0000","last_became_peered":"2023-07-12T15:27:46.169282+0000","last_unstale":"2023-07-12T15:27:50.166336+0000","last_undegraded":"2023-07-12T15:27:50.166336+0000","last_fullsized":"2023-07-12T15:27:50.166336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:06:20.633426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167179+0000","last_change":"2023-07-12T15:27:44.152244+0000","last_active":"2023-07-12T15:27:50.167179+0000","last_peered":"2023-07-12T15:27:50.167179+0000","last_clean":"2023-07-12T15:27:50.167179+0000","last_became_active":"2023-07-12T15:27:44.151851+0000","last_became_peered":"2023-07-12T15:27:44.151851+0000","last_unstale":"2023-07-12T15:27:50.167179+0000","last_undegraded":"2023-07-12T15:27:50.167179+0000","last_fullsized":"2023-07-12T15:27:50.167179+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:32:30.762564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166212+0000","last_change":"2023-07-12T15:27:40.130987+0000","last_active":"2023-07-12T15:27:50.166212+0000","last_peered":"2023-07-12T15:27:50.166212+0000","last_clean":"2023-07-12T15:27:50.166212+0000","last_became_active":"2023-07-12T15:27:40.130599+0000","last_became_peered":"2023-07-12T15:27:40.130599+0000","last_unstale":"2023-07-12T15:27:50.166212+0000","last_undegraded":"2023-07-12T15:27:50.166212+0000","last_fullsized":"2023-07-12T15:27:50.166212+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T15:28:30.296076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954687+0000","last_change":"2023-07-12T15:27:42.128982+0000","last_active":"2023-07-12T15:27:49.954687+0000","last_peered":"2023-07-12T15:27:49.954687+0000","last_clean":"2023-07-12T15:27:49.954687+0000","last_became_active":"2023-07-12T15:27:42.128687+0000","last_became_peered":"2023-07-12T15:27:42.128687+0000","last_unstale":"2023-07-12T15:27:49.954687+0000","last_undegraded":"2023-07-12T15:27:49.954687+0000","last_fullsized":"2023-07-12T15:27:49.954687+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:16:49.488081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.168258+0000","last_change":"2023-07-12T15:27:46.161337+0000","last_active":"2023-07-12T15:27:50.168258+0000","last_peered":"2023-07-12T15:27:50.168258+0000","last_clean":"2023-07-12T15:27:50.168258+0000","last_became_active":"2023-07-12T15:27:46.161070+0000","last_became_peered":"2023-07-12T15:27:46.161070+0000","last_unstale":"2023-07-12T15:27:50.168258+0000","last_undegraded":"2023-07-12T15:27:50.168258+0000","last_fullsized":"2023-07-12T15:27:50.168258+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:20:31.063627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'11","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:28.814172+0000","last_change":"2023-07-12T15:27:44.151955+0000","last_active":"2023-07-12T15:28:28.814172+0000","last_peered":"2023-07-12T15:28:28.814172+0000","last_clean":"2023-07-12T15:28:28.814172+0000","last_became_active":"2023-07-12T15:27:44.151546+0000","last_became_peered":"2023-07-12T15:27:44.151546+0000","last_unstale":"2023-07-12T15:28:28.814172+0000","last_undegraded":"2023-07-12T15:28:28.814172+0000","last_fullsized":"2023-07-12T15:28:28.814172+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:04:19.255266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165972+0000","last_change":"2023-07-12T15:27:40.122947+0000","last_active":"2023-07-12T15:27:50.165972+0000","last_peered":"2023-07-12T15:27:50.165972+0000","last_clean":"2023-07-12T15:27:50.165972+0000","last_became_active":"2023-07-12T15:27:40.122685+0000","last_became_peered":"2023-07-12T15:27:40.122685+0000","last_unstale":"2023-07-12T15:27:50.165972+0000","last_undegraded":"2023-07-12T15:27:50.165972+0000","last_fullsized":"2023-07-12T15:27:50.165972+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:21:21.256101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952832+0000","last_change":"2023-07-12T15:27:42.131173+0000","last_active":"2023-07-12T15:27:49.952832+0000","last_peered":"2023-07-12T15:27:49.952832+0000","last_clean":"2023-07-12T15:27:49.952832+0000","last_became_active":"2023-07-12T15:27:42.130880+0000","last_became_peered":"2023-07-12T15:27:42.130880+0000","last_unstale":"2023-07-12T15:27:49.952832+0000","last_undegraded":"2023-07-12T15:27:49.952832+0000","last_fullsized":"2023-07-12T15:27:49.952832+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:06:14.663877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166387+0000","last_change":"2023-07-12T15:27:46.168899+0000","last_active":"2023-07-12T15:27:50.166387+0000","last_peered":"2023-07-12T15:27:50.166387+0000","last_clean":"2023-07-12T15:27:50.166387+0000","last_became_active":"2023-07-12T15:27:46.168511+0000","last_became_peered":"2023-07-12T15:27:46.168511+0000","last_unstale":"2023-07-12T15:27:50.166387+0000","last_undegraded":"2023-07-12T15:27:50.166387+0000","last_fullsized":"2023-07-12T15:27:50.166387+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:28:38.893034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'11","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:28.814394+0000","last_change":"2023-07-12T15:27:44.147818+0000","last_active":"2023-07-12T15:28:28.814394+0000","last_peered":"2023-07-12T15:28:28.814394+0000","last_clean":"2023-07-12T15:28:28.814394+0000","last_became_active":"2023-07-12T15:27:44.147589+0000","last_became_peered":"2023-07-12T15:27:44.147589+0000","last_unstale":"2023-07-12T15:28:28.814394+0000","last_undegraded":"2023-07-12T15:28:28.814394+0000","last_fullsized":"2023-07-12T15:28:28.814394+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:56:50.718828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952389+0000","last_change":"2023-07-12T15:27:40.134832+0000","last_active":"2023-07-12T15:27:49.952389+0000","last_peered":"2023-07-12T15:27:49.952389+0000","last_clean":"2023-07-12T15:27:49.952389+0000","last_became_active":"2023-07-12T15:27:40.134509+0000","last_became_peered":"2023-07-12T15:27:40.134509+0000","last_unstale":"2023-07-12T15:27:49.952389+0000","last_undegraded":"2023-07-12T15:27:49.952389+0000","last_fullsized":"2023-07-12T15:27:49.952389+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:02:32.614487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165547+0000","last_change":"2023-07-12T15:27:42.133184+0000","last_active":"2023-07-12T15:27:50.165547+0000","last_peered":"2023-07-12T15:27:50.165547+0000","last_clean":"2023-07-12T15:27:50.165547+0000","last_became_active":"2023-07-12T15:27:42.132932+0000","last_became_peered":"2023-07-12T15:27:42.132932+0000","last_unstale":"2023-07-12T15:27:50.165547+0000","last_undegraded":"2023-07-12T15:27:50.165547+0000","last_fullsized":"2023-07-12T15:27:50.165547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:25:11.858527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167709+0000","last_change":"2023-07-12T15:27:46.718543+0000","last_active":"2023-07-12T15:27:50.167709+0000","last_peered":"2023-07-12T15:27:50.167709+0000","last_clean":"2023-07-12T15:27:50.167709+0000","last_became_active":"2023-07-12T15:27:46.717706+0000","last_became_peered":"2023-07-12T15:27:46.717706+0000","last_unstale":"2023-07-12T15:27:50.167709+0000","last_undegraded":"2023-07-12T15:27:50.167709+0000","last_fullsized":"2023-07-12T15:27:50.167709+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:57:35.384154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952721+0000","last_change":"2023-07-12T15:27:44.151747+0000","last_active":"2023-07-12T15:27:49.952721+0000","last_peered":"2023-07-12T15:27:49.952721+0000","last_clean":"2023-07-12T15:27:49.952721+0000","last_became_active":"2023-07-12T15:27:44.151415+0000","last_became_peered":"2023-07-12T15:27:44.151415+0000","last_unstale":"2023-07-12T15:27:49.952721+0000","last_undegraded":"2023-07-12T15:27:49.952721+0000","last_fullsized":"2023-07-12T15:27:49.952721+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:53:14.702529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167707+0000","last_change":"2023-07-12T15:27:40.124291+0000","last_active":"2023-07-12T15:27:50.167707+0000","last_peered":"2023-07-12T15:27:50.167707+0000","last_clean":"2023-07-12T15:27:50.167707+0000","last_became_active":"2023-07-12T15:27:40.123913+0000","last_became_peered":"2023-07-12T15:27:40.123913+0000","last_unstale":"2023-07-12T15:27:50.167707+0000","last_undegraded":"2023-07-12T15:27:50.167707+0000","last_fullsized":"2023-07-12T15:27:50.167707+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:05:33.567531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":26,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165602+0000","last_change":"2023-07-12T15:27:42.132617+0000","last_active":"2023-07-12T15:27:50.165602+0000","last_peered":"2023-07-12T15:27:50.165602+0000","last_clean":"2023-07-12T15:27:50.165602+0000","last_became_active":"2023-07-12T15:27:42.132327+0000","last_became_peered":"2023-07-12T15:27:42.132327+0000","last_unstale":"2023-07-12T15:27:50.165602+0000","last_undegraded":"2023-07-12T15:27:50.165602+0000","last_fullsized":"2023-07-12T15:27:50.165602+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:40:47.844771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955866+0000","last_change":"2023-07-12T15:27:46.161301+0000","last_active":"2023-07-12T15:27:49.955866+0000","last_peered":"2023-07-12T15:27:49.955866+0000","last_clean":"2023-07-12T15:27:49.955866+0000","last_became_active":"2023-07-12T15:27:46.160999+0000","last_became_peered":"2023-07-12T15:27:46.160999+0000","last_unstale":"2023-07-12T15:27:49.955866+0000","last_undegraded":"2023-07-12T15:27:49.955866+0000","last_fullsized":"2023-07-12T15:27:49.955866+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:02:24.974403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167053+0000","last_change":"2023-07-12T15:27:44.151957+0000","last_active":"2023-07-12T15:27:50.167053+0000","last_peered":"2023-07-12T15:27:50.167053+0000","last_clean":"2023-07-12T15:27:50.167053+0000","last_became_active":"2023-07-12T15:27:44.151629+0000","last_became_peered":"2023-07-12T15:27:44.151629+0000","last_unstale":"2023-07-12T15:27:50.167053+0000","last_undegraded":"2023-07-12T15:27:50.167053+0000","last_fullsized":"2023-07-12T15:27:50.167053+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:27:28.676471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955867+0000","last_change":"2023-07-12T15:27:40.124203+0000","last_active":"2023-07-12T15:27:49.955867+0000","last_peered":"2023-07-12T15:27:49.955867+0000","last_clean":"2023-07-12T15:27:49.955867+0000","last_became_active":"2023-07-12T15:27:40.123980+0000","last_became_peered":"2023-07-12T15:27:40.123980+0000","last_unstale":"2023-07-12T15:27:49.955867+0000","last_undegraded":"2023-07-12T15:27:49.955867+0000","last_fullsized":"2023-07-12T15:27:49.955867+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:26:07.683170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166504+0000","last_change":"2023-07-12T15:27:42.136284+0000","last_active":"2023-07-12T15:27:50.166504+0000","last_peered":"2023-07-12T15:27:50.166504+0000","last_clean":"2023-07-12T15:27:50.166504+0000","last_became_active":"2023-07-12T15:27:42.135687+0000","last_became_peered":"2023-07-12T15:27:42.135687+0000","last_unstale":"2023-07-12T15:27:50.166504+0000","last_undegraded":"2023-07-12T15:27:50.166504+0000","last_fullsized":"2023-07-12T15:27:50.166504+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:18:59.390617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167282+0000","last_change":"2023-07-12T15:27:46.165713+0000","last_active":"2023-07-12T15:27:50.167282+0000","last_peered":"2023-07-12T15:27:50.167282+0000","last_clean":"2023-07-12T15:27:50.167282+0000","last_became_active":"2023-07-12T15:27:46.165313+0000","last_became_peered":"2023-07-12T15:27:46.165313+0000","last_unstale":"2023-07-12T15:27:50.167282+0000","last_undegraded":"2023-07-12T15:27:50.167282+0000","last_fullsized":"2023-07-12T15:27:50.167282+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:22:47.525502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"60'2","reported_seq":28,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.168012+0000","last_change":"2023-07-12T15:27:40.131073+0000","last_active":"2023-07-12T15:27:50.168012+0000","last_peered":"2023-07-12T15:27:50.168012+0000","last_clean":"2023-07-12T15:27:50.168012+0000","last_became_active":"2023-07-12T15:27:40.130729+0000","last_became_peered":"2023-07-12T15:27:40.130729+0000","last_unstale":"2023-07-12T15:27:50.168012+0000","last_undegraded":"2023-07-12T15:27:50.168012+0000","last_fullsized":"2023-07-12T15:27:50.168012+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:51:07.946411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":40,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167422+0000","last_change":"2023-07-12T15:27:42.140164+0000","last_active":"2023-07-12T15:27:50.167422+0000","last_peered":"2023-07-12T15:27:50.167422+0000","last_clean":"2023-07-12T15:27:50.167422+0000","last_became_active":"2023-07-12T15:27:42.139824+0000","last_became_peered":"2023-07-12T15:27:42.139824+0000","last_unstale":"2023-07-12T15:27:50.167422+0000","last_undegraded":"2023-07-12T15:27:50.167422+0000","last_fullsized":"2023-07-12T15:27:50.167422+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:30:30.583175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955833+0000","last_change":"2023-07-12T15:27:44.144671+0000","last_active":"2023-07-12T15:27:49.955833+0000","last_peered":"2023-07-12T15:27:49.955833+0000","last_clean":"2023-07-12T15:27:49.955833+0000","last_became_active":"2023-07-12T15:27:44.143852+0000","last_became_peered":"2023-07-12T15:27:44.143852+0000","last_unstale":"2023-07-12T15:27:49.955833+0000","last_undegraded":"2023-07-12T15:27:49.955833+0000","last_fullsized":"2023-07-12T15:27:49.955833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:32:53.808907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166542+0000","last_change":"2023-07-12T15:27:46.169482+0000","last_active":"2023-07-12T15:27:50.166542+0000","last_peered":"2023-07-12T15:27:50.166542+0000","last_clean":"2023-07-12T15:27:50.166542+0000","last_became_active":"2023-07-12T15:27:46.168837+0000","last_became_peered":"2023-07-12T15:27:46.168837+0000","last_unstale":"2023-07-12T15:27:50.166542+0000","last_undegraded":"2023-07-12T15:27:50.166542+0000","last_fullsized":"2023-07-12T15:27:50.166542+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:49:10.960844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955323+0000","last_change":"2023-07-12T15:27:40.129384+0000","last_active":"2023-07-12T15:27:49.955323+0000","last_peered":"2023-07-12T15:27:49.955323+0000","last_clean":"2023-07-12T15:27:49.955323+0000","last_became_active":"2023-07-12T15:27:40.129107+0000","last_became_peered":"2023-07-12T15:27:40.129107+0000","last_unstale":"2023-07-12T15:27:49.955323+0000","last_undegraded":"2023-07-12T15:27:49.955323+0000","last_fullsized":"2023-07-12T15:27:49.955323+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:21:16.040857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954842+0000","last_change":"2023-07-12T15:27:42.132498+0000","last_active":"2023-07-12T15:27:49.954842+0000","last_peered":"2023-07-12T15:27:49.954842+0000","last_clean":"2023-07-12T15:27:49.954842+0000","last_became_active":"2023-07-12T15:27:42.132241+0000","last_became_peered":"2023-07-12T15:27:42.132241+0000","last_unstale":"2023-07-12T15:27:49.954842+0000","last_undegraded":"2023-07-12T15:27:49.954842+0000","last_fullsized":"2023-07-12T15:27:49.954842+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:39:34.735107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'11","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:28.814111+0000","last_change":"2023-07-12T15:27:44.155124+0000","last_active":"2023-07-12T15:28:28.814111+0000","last_peered":"2023-07-12T15:28:28.814111+0000","last_clean":"2023-07-12T15:28:28.814111+0000","last_became_active":"2023-07-12T15:27:44.154627+0000","last_became_peered":"2023-07-12T15:27:44.154627+0000","last_unstale":"2023-07-12T15:28:28.814111+0000","last_undegraded":"2023-07-12T15:28:28.814111+0000","last_fullsized":"2023-07-12T15:28:28.814111+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T20:37:59.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166896+0000","last_change":"2023-07-12T15:27:46.164687+0000","last_active":"2023-07-12T15:27:50.166896+0000","last_peered":"2023-07-12T15:27:50.166896+0000","last_clean":"2023-07-12T15:27:50.166896+0000","last_became_active":"2023-07-12T15:27:46.163875+0000","last_became_peered":"2023-07-12T15:27:46.163875+0000","last_unstale":"2023-07-12T15:27:50.166896+0000","last_undegraded":"2023-07-12T15:27:50.166896+0000","last_fullsized":"2023-07-12T15:27:50.166896+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:36:52.918069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952431+0000","last_change":"2023-07-12T15:27:40.130331+0000","last_active":"2023-07-12T15:27:49.952431+0000","last_peered":"2023-07-12T15:27:49.952431+0000","last_clean":"2023-07-12T15:27:49.952431+0000","last_became_active":"2023-07-12T15:27:40.130019+0000","last_became_peered":"2023-07-12T15:27:40.130019+0000","last_unstale":"2023-07-12T15:27:49.952431+0000","last_undegraded":"2023-07-12T15:27:49.952431+0000","last_fullsized":"2023-07-12T15:27:49.952431+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:18:47.396545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'30","reported_seq":79,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:28.814121+0000","last_change":"2023-07-12T15:27:42.133394+0000","last_active":"2023-07-12T15:28:28.814121+0000","last_peered":"2023-07-12T15:28:28.814121+0000","last_clean":"2023-07-12T15:28:28.814121+0000","last_became_active":"2023-07-12T15:27:42.133073+0000","last_became_peered":"2023-07-12T15:27:42.133073+0000","last_unstale":"2023-07-12T15:28:28.814121+0000","last_undegraded":"2023-07-12T15:28:28.814121+0000","last_fullsized":"2023-07-12T15:28:28.814121+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T15:33:49.771223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166475+0000","last_change":"2023-07-12T15:27:44.151055+0000","last_active":"2023-07-12T15:27:50.166475+0000","last_peered":"2023-07-12T15:27:50.166475+0000","last_clean":"2023-07-12T15:27:50.166475+0000","last_became_active":"2023-07-12T15:27:44.150748+0000","last_became_peered":"2023-07-12T15:27:44.150748+0000","last_unstale":"2023-07-12T15:27:50.166475+0000","last_undegraded":"2023-07-12T15:27:50.166475+0000","last_fullsized":"2023-07-12T15:27:50.166475+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:10:09.409726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167891+0000","last_change":"2023-07-12T15:27:46.717308+0000","last_active":"2023-07-12T15:27:50.167891+0000","last_peered":"2023-07-12T15:27:50.167891+0000","last_clean":"2023-07-12T15:27:50.167891+0000","last_became_active":"2023-07-12T15:27:46.717055+0000","last_became_peered":"2023-07-12T15:27:46.717055+0000","last_unstale":"2023-07-12T15:27:50.167891+0000","last_undegraded":"2023-07-12T15:27:50.167891+0000","last_fullsized":"2023-07-12T15:27:50.167891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:43:34.142083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"68'5","reported_seq":75,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:32.925479+0000","last_change":"2023-07-12T15:27:45.145007+0000","last_active":"2023-07-12T15:28:32.925479+0000","last_peered":"2023-07-12T15:28:32.925479+0000","last_clean":"2023-07-12T15:28:32.925479+0000","last_became_active":"2023-07-12T15:27:41.130547+0000","last_became_peered":"2023-07-12T15:27:41.130547+0000","last_unstale":"2023-07-12T15:28:32.925479+0000","last_undegraded":"2023-07-12T15:28:32.925479+0000","last_fullsized":"2023-07-12T15:28:32.925479+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:40.106345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:40.106345+0000","last_clean_scrub_stamp":"2023-07-12T15:27:40.106345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:51:09.700281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034162100000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":49,"num_read_kb":44,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955706+0000","last_change":"2023-07-12T15:27:40.124197+0000","last_active":"2023-07-12T15:27:49.955706+0000","last_peered":"2023-07-12T15:27:49.955706+0000","last_clean":"2023-07-12T15:27:49.955706+0000","last_became_active":"2023-07-12T15:27:40.123876+0000","last_became_peered":"2023-07-12T15:27:40.123876+0000","last_unstale":"2023-07-12T15:27:49.955706+0000","last_undegraded":"2023-07-12T15:27:49.955706+0000","last_fullsized":"2023-07-12T15:27:49.955706+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:41:34.484752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":44,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956805+0000","last_change":"2023-07-12T15:27:42.133473+0000","last_active":"2023-07-12T15:27:49.956805+0000","last_peered":"2023-07-12T15:27:49.956805+0000","last_clean":"2023-07-12T15:27:49.956805+0000","last_became_active":"2023-07-12T15:27:42.133122+0000","last_became_peered":"2023-07-12T15:27:42.133122+0000","last_unstale":"2023-07-12T15:27:49.956805+0000","last_undegraded":"2023-07-12T15:27:49.956805+0000","last_fullsized":"2023-07-12T15:27:49.956805+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:13:29.009977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956114+0000","last_change":"2023-07-12T15:27:44.153052+0000","last_active":"2023-07-12T15:27:49.956114+0000","last_peered":"2023-07-12T15:27:49.956114+0000","last_clean":"2023-07-12T15:27:49.956114+0000","last_became_active":"2023-07-12T15:27:44.152754+0000","last_became_peered":"2023-07-12T15:27:44.152754+0000","last_unstale":"2023-07-12T15:27:49.956114+0000","last_undegraded":"2023-07-12T15:27:49.956114+0000","last_fullsized":"2023-07-12T15:27:49.956114+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:40:57.401230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.953269+0000","last_change":"2023-07-12T15:27:46.165625+0000","last_active":"2023-07-12T15:27:49.953269+0000","last_peered":"2023-07-12T15:27:49.953269+0000","last_clean":"2023-07-12T15:27:49.953269+0000","last_became_active":"2023-07-12T15:27:46.165341+0000","last_became_peered":"2023-07-12T15:27:46.165341+0000","last_unstale":"2023-07-12T15:27:49.953269+0000","last_undegraded":"2023-07-12T15:27:49.953269+0000","last_fullsized":"2023-07-12T15:27:49.953269+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:30:55.751614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"65'2","reported_seq":25,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167125+0000","last_change":"2023-07-12T15:27:45.143925+0000","last_active":"2023-07-12T15:27:50.167125+0000","last_peered":"2023-07-12T15:27:50.167125+0000","last_clean":"2023-07-12T15:27:50.167125+0000","last_became_active":"2023-07-12T15:27:41.130428+0000","last_became_peered":"2023-07-12T15:27:41.130428+0000","last_unstale":"2023-07-12T15:27:50.167125+0000","last_undegraded":"2023-07-12T15:27:50.167125+0000","last_fullsized":"2023-07-12T15:27:50.167125+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:40.106345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:40.106345+0000","last_clean_scrub_stamp":"2023-07-12T15:27:40.106345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:12:37.320950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065463700000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952535+0000","last_change":"2023-07-12T15:27:40.133053+0000","last_active":"2023-07-12T15:27:49.952535+0000","last_peered":"2023-07-12T15:27:49.952535+0000","last_clean":"2023-07-12T15:27:49.952535+0000","last_became_active":"2023-07-12T15:27:40.132735+0000","last_became_peered":"2023-07-12T15:27:40.132735+0000","last_unstale":"2023-07-12T15:27:49.952535+0000","last_undegraded":"2023-07-12T15:27:49.952535+0000","last_fullsized":"2023-07-12T15:27:49.952535+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:38:46.103567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"69'128","reported_seq":266,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:13.490058+0000","last_change":"2023-07-12T15:27:30.563173+0000","last_active":"2023-07-12T15:28:13.490058+0000","last_peered":"2023-07-12T15:28:13.490058+0000","last_clean":"2023-07-12T15:28:13.490058+0000","last_became_active":"2023-07-12T15:27:30.562668+0000","last_became_peered":"2023-07-12T15:27:30.562668+0000","last_unstale":"2023-07-12T15:28:13.490058+0000","last_undegraded":"2023-07-12T15:28:13.490058+0000","last_fullsized":"2023-07-12T15:28:13.490058+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:25:40.483044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:25:40.483044+0000","last_clean_scrub_stamp":"2023-07-12T15:25:40.483044+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:26:28.800483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"67'14","reported_seq":39,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166506+0000","last_change":"2023-07-12T15:27:42.132132+0000","last_active":"2023-07-12T15:27:50.166506+0000","last_peered":"2023-07-12T15:27:50.166506+0000","last_clean":"2023-07-12T15:27:50.166506+0000","last_became_active":"2023-07-12T15:27:42.131862+0000","last_became_peered":"2023-07-12T15:27:42.131862+0000","last_unstale":"2023-07-12T15:27:50.166506+0000","last_undegraded":"2023-07-12T15:27:50.166506+0000","last_fullsized":"2023-07-12T15:27:50.166506+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:02:26.304626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166122+0000","last_change":"2023-07-12T15:27:44.157770+0000","last_active":"2023-07-12T15:27:50.166122+0000","last_peered":"2023-07-12T15:27:50.166122+0000","last_clean":"2023-07-12T15:27:50.166122+0000","last_became_active":"2023-07-12T15:27:44.157494+0000","last_became_peered":"2023-07-12T15:27:44.157494+0000","last_unstale":"2023-07-12T15:27:50.166122+0000","last_undegraded":"2023-07-12T15:27:50.166122+0000","last_fullsized":"2023-07-12T15:27:50.166122+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:51:16.040821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954959+0000","last_change":"2023-07-12T15:27:46.170137+0000","last_active":"2023-07-12T15:27:49.954959+0000","last_peered":"2023-07-12T15:27:49.954959+0000","last_clean":"2023-07-12T15:27:49.954959+0000","last_became_active":"2023-07-12T15:27:46.169855+0000","last_became_peered":"2023-07-12T15:27:46.169855+0000","last_unstale":"2023-07-12T15:27:49.954959+0000","last_undegraded":"2023-07-12T15:27:49.954959+0000","last_fullsized":"2023-07-12T15:27:49.954959+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:44:52.967353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166176+0000","last_change":"2023-07-12T15:27:45.146062+0000","last_active":"2023-07-12T15:27:50.166176+0000","last_peered":"2023-07-12T15:27:50.166176+0000","last_clean":"2023-07-12T15:27:50.166176+0000","last_became_active":"2023-07-12T15:27:41.128841+0000","last_became_peered":"2023-07-12T15:27:41.128841+0000","last_unstale":"2023-07-12T15:27:50.166176+0000","last_undegraded":"2023-07-12T15:27:50.166176+0000","last_fullsized":"2023-07-12T15:27:50.166176+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:40.106345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:40.106345+0000","last_clean_scrub_stamp":"2023-07-12T15:27:40.106345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:56:51.194915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075130800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.168153+0000","last_change":"2023-07-12T15:27:40.130214+0000","last_active":"2023-07-12T15:27:50.168153+0000","last_peered":"2023-07-12T15:27:50.168153+0000","last_clean":"2023-07-12T15:27:50.168153+0000","last_became_active":"2023-07-12T15:27:40.129920+0000","last_became_peered":"2023-07-12T15:27:40.129920+0000","last_unstale":"2023-07-12T15:27:50.168153+0000","last_undegraded":"2023-07-12T15:27:50.168153+0000","last_fullsized":"2023-07-12T15:27:50.168153+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:03:47.596959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":35,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166221+0000","last_change":"2023-07-12T15:27:42.138281+0000","last_active":"2023-07-12T15:27:50.166221+0000","last_peered":"2023-07-12T15:27:50.166221+0000","last_clean":"2023-07-12T15:27:50.166221+0000","last_became_active":"2023-07-12T15:27:42.137694+0000","last_became_peered":"2023-07-12T15:27:42.137694+0000","last_unstale":"2023-07-12T15:27:50.166221+0000","last_undegraded":"2023-07-12T15:27:50.166221+0000","last_fullsized":"2023-07-12T15:27:50.166221+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:12:33.813764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.953196+0000","last_change":"2023-07-12T15:27:44.151774+0000","last_active":"2023-07-12T15:27:49.953196+0000","last_peered":"2023-07-12T15:27:49.953196+0000","last_clean":"2023-07-12T15:27:49.953196+0000","last_became_active":"2023-07-12T15:27:44.151468+0000","last_became_peered":"2023-07-12T15:27:44.151468+0000","last_unstale":"2023-07-12T15:27:49.953196+0000","last_undegraded":"2023-07-12T15:27:49.953196+0000","last_fullsized":"2023-07-12T15:27:49.953196+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:21:22.532171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954408+0000","last_change":"2023-07-12T15:27:46.717552+0000","last_active":"2023-07-12T15:27:49.954408+0000","last_peered":"2023-07-12T15:27:49.954408+0000","last_clean":"2023-07-12T15:27:49.954408+0000","last_became_active":"2023-07-12T15:27:46.717184+0000","last_became_peered":"2023-07-12T15:27:46.717184+0000","last_unstale":"2023-07-12T15:27:49.954408+0000","last_undegraded":"2023-07-12T15:27:49.954408+0000","last_fullsized":"2023-07-12T15:27:49.954408+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:53:42.636521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167938+0000","last_change":"2023-07-12T15:27:40.135385+0000","last_active":"2023-07-12T15:27:50.167938+0000","last_peered":"2023-07-12T15:27:50.167938+0000","last_clean":"2023-07-12T15:27:50.167938+0000","last_became_active":"2023-07-12T15:27:40.134949+0000","last_became_peered":"2023-07-12T15:27:40.134949+0000","last_unstale":"2023-07-12T15:27:50.167938+0000","last_undegraded":"2023-07-12T15:27:50.167938+0000","last_fullsized":"2023-07-12T15:27:50.167938+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:05:07.598634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":53,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165547+0000","last_change":"2023-07-12T15:27:42.134809+0000","last_active":"2023-07-12T15:27:50.165547+0000","last_peered":"2023-07-12T15:27:50.165547+0000","last_clean":"2023-07-12T15:27:50.165547+0000","last_became_active":"2023-07-12T15:27:42.134514+0000","last_became_peered":"2023-07-12T15:27:42.134514+0000","last_unstale":"2023-07-12T15:27:50.165547+0000","last_undegraded":"2023-07-12T15:27:50.165547+0000","last_fullsized":"2023-07-12T15:27:50.165547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:54:12.844302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166698+0000","last_change":"2023-07-12T15:27:44.154840+0000","last_active":"2023-07-12T15:27:50.166698+0000","last_peered":"2023-07-12T15:27:50.166698+0000","last_clean":"2023-07-12T15:27:50.166698+0000","last_became_active":"2023-07-12T15:27:44.154537+0000","last_became_peered":"2023-07-12T15:27:44.154537+0000","last_unstale":"2023-07-12T15:27:50.166698+0000","last_undegraded":"2023-07-12T15:27:50.166698+0000","last_fullsized":"2023-07-12T15:27:50.166698+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:27:07.097195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166786+0000","last_change":"2023-07-12T15:27:46.163445+0000","last_active":"2023-07-12T15:27:50.166786+0000","last_peered":"2023-07-12T15:27:50.166786+0000","last_clean":"2023-07-12T15:27:50.166786+0000","last_became_active":"2023-07-12T15:27:46.162794+0000","last_became_peered":"2023-07-12T15:27:46.162794+0000","last_unstale":"2023-07-12T15:27:50.166786+0000","last_undegraded":"2023-07-12T15:27:50.166786+0000","last_fullsized":"2023-07-12T15:27:50.166786+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T15:51:01.966202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956429+0000","last_change":"2023-07-12T15:27:40.128114+0000","last_active":"2023-07-12T15:27:49.956429+0000","last_peered":"2023-07-12T15:27:49.956429+0000","last_clean":"2023-07-12T15:27:49.956429+0000","last_became_active":"2023-07-12T15:27:40.127814+0000","last_became_peered":"2023-07-12T15:27:40.127814+0000","last_unstale":"2023-07-12T15:27:49.956429+0000","last_undegraded":"2023-07-12T15:27:49.956429+0000","last_fullsized":"2023-07-12T15:27:49.956429+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:58:48.765074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":49,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165935+0000","last_change":"2023-07-12T15:27:42.134764+0000","last_active":"2023-07-12T15:27:50.165935+0000","last_peered":"2023-07-12T15:27:50.165935+0000","last_clean":"2023-07-12T15:27:50.165935+0000","last_became_active":"2023-07-12T15:27:42.134517+0000","last_became_peered":"2023-07-12T15:27:42.134517+0000","last_unstale":"2023-07-12T15:27:50.165935+0000","last_undegraded":"2023-07-12T15:27:50.165935+0000","last_fullsized":"2023-07-12T15:27:50.165935+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:15:04.623968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167876+0000","last_change":"2023-07-12T15:27:44.152178+0000","last_active":"2023-07-12T15:27:50.167876+0000","last_peered":"2023-07-12T15:27:50.167876+0000","last_clean":"2023-07-12T15:27:50.167876+0000","last_became_active":"2023-07-12T15:27:44.151878+0000","last_became_peered":"2023-07-12T15:27:44.151878+0000","last_unstale":"2023-07-12T15:27:50.167876+0000","last_undegraded":"2023-07-12T15:27:50.167876+0000","last_fullsized":"2023-07-12T15:27:50.167876+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:49:21.080732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167501+0000","last_change":"2023-07-12T15:27:46.720176+0000","last_active":"2023-07-12T15:27:50.167501+0000","last_peered":"2023-07-12T15:27:50.167501+0000","last_clean":"2023-07-12T15:27:50.167501+0000","last_became_active":"2023-07-12T15:27:46.719760+0000","last_became_peered":"2023-07-12T15:27:46.719760+0000","last_unstale":"2023-07-12T15:27:50.167501+0000","last_undegraded":"2023-07-12T15:27:50.167501+0000","last_fullsized":"2023-07-12T15:27:50.167501+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:00:41.076133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167152+0000","last_change":"2023-07-12T15:27:40.129650+0000","last_active":"2023-07-12T15:27:50.167152+0000","last_peered":"2023-07-12T15:27:50.167152+0000","last_clean":"2023-07-12T15:27:50.167152+0000","last_became_active":"2023-07-12T15:27:40.129241+0000","last_became_peered":"2023-07-12T15:27:40.129241+0000","last_unstale":"2023-07-12T15:27:50.167152+0000","last_undegraded":"2023-07-12T15:27:50.167152+0000","last_fullsized":"2023-07-12T15:27:50.167152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:23:17.132392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":38,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167656+0000","last_change":"2023-07-12T15:27:42.138593+0000","last_active":"2023-07-12T15:27:50.167656+0000","last_peered":"2023-07-12T15:27:50.167656+0000","last_clean":"2023-07-12T15:27:50.167656+0000","last_became_active":"2023-07-12T15:27:42.138275+0000","last_became_peered":"2023-07-12T15:27:42.138275+0000","last_unstale":"2023-07-12T15:27:50.167656+0000","last_undegraded":"2023-07-12T15:27:50.167656+0000","last_fullsized":"2023-07-12T15:27:50.167656+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:28:49.929423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'11","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:28.814169+0000","last_change":"2023-07-12T15:27:44.155072+0000","last_active":"2023-07-12T15:28:28.814169+0000","last_peered":"2023-07-12T15:28:28.814169+0000","last_clean":"2023-07-12T15:28:28.814169+0000","last_became_active":"2023-07-12T15:27:44.154800+0000","last_became_peered":"2023-07-12T15:27:44.154800+0000","last_unstale":"2023-07-12T15:28:28.814169+0000","last_undegraded":"2023-07-12T15:28:28.814169+0000","last_fullsized":"2023-07-12T15:28:28.814169+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:25:04.242576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166978+0000","last_change":"2023-07-12T15:27:46.718096+0000","last_active":"2023-07-12T15:27:50.166978+0000","last_peered":"2023-07-12T15:27:50.166978+0000","last_clean":"2023-07-12T15:27:50.166978+0000","last_became_active":"2023-07-12T15:27:46.717778+0000","last_became_peered":"2023-07-12T15:27:46.717778+0000","last_unstale":"2023-07-12T15:27:50.166978+0000","last_undegraded":"2023-07-12T15:27:50.166978+0000","last_fullsized":"2023-07-12T15:27:50.166978+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:54:33.593328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.953066+0000","last_change":"2023-07-12T15:27:44.153649+0000","last_active":"2023-07-12T15:27:49.953066+0000","last_peered":"2023-07-12T15:27:49.953066+0000","last_clean":"2023-07-12T15:27:49.953066+0000","last_became_active":"2023-07-12T15:27:44.153297+0000","last_became_peered":"2023-07-12T15:27:44.153297+0000","last_unstale":"2023-07-12T15:27:49.953066+0000","last_undegraded":"2023-07-12T15:27:49.953066+0000","last_fullsized":"2023-07-12T15:27:49.953066+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:40:58.363613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.168272+0000","last_change":"2023-07-12T15:27:40.130023+0000","last_active":"2023-07-12T15:27:50.168272+0000","last_peered":"2023-07-12T15:27:50.168272+0000","last_clean":"2023-07-12T15:27:50.168272+0000","last_became_active":"2023-07-12T15:27:40.129662+0000","last_became_peered":"2023-07-12T15:27:40.129662+0000","last_unstale":"2023-07-12T15:27:50.168272+0000","last_undegraded":"2023-07-12T15:27:50.168272+0000","last_fullsized":"2023-07-12T15:27:50.168272+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:46:44.335318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166796+0000","last_change":"2023-07-12T15:27:42.140160+0000","last_active":"2023-07-12T15:27:50.166796+0000","last_peered":"2023-07-12T15:27:50.166796+0000","last_clean":"2023-07-12T15:27:50.166796+0000","last_became_active":"2023-07-12T15:27:42.139927+0000","last_became_peered":"2023-07-12T15:27:42.139927+0000","last_unstale":"2023-07-12T15:27:50.166796+0000","last_undegraded":"2023-07-12T15:27:50.166796+0000","last_fullsized":"2023-07-12T15:27:50.166796+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:42:04.716802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952668+0000","last_change":"2023-07-12T15:27:46.166865+0000","last_active":"2023-07-12T15:27:49.952668+0000","last_peered":"2023-07-12T15:27:49.952668+0000","last_clean":"2023-07-12T15:27:49.952668+0000","last_became_active":"2023-07-12T15:27:46.166552+0000","last_became_peered":"2023-07-12T15:27:46.166552+0000","last_unstale":"2023-07-12T15:27:49.952668+0000","last_undegraded":"2023-07-12T15:27:49.952668+0000","last_fullsized":"2023-07-12T15:27:49.952668+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:48:27.558516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'11","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:28.814393+0000","last_change":"2023-07-12T15:27:44.148480+0000","last_active":"2023-07-12T15:28:28.814393+0000","last_peered":"2023-07-12T15:28:28.814393+0000","last_clean":"2023-07-12T15:28:28.814393+0000","last_became_active":"2023-07-12T15:27:44.148136+0000","last_became_peered":"2023-07-12T15:27:44.148136+0000","last_unstale":"2023-07-12T15:28:28.814393+0000","last_undegraded":"2023-07-12T15:28:28.814393+0000","last_fullsized":"2023-07-12T15:28:28.814393+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:28:53.776463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954638+0000","last_change":"2023-07-12T15:27:40.129260+0000","last_active":"2023-07-12T15:27:49.954638+0000","last_peered":"2023-07-12T15:27:49.954638+0000","last_clean":"2023-07-12T15:27:49.954638+0000","last_became_active":"2023-07-12T15:27:40.129004+0000","last_became_peered":"2023-07-12T15:27:40.129004+0000","last_unstale":"2023-07-12T15:27:49.954638+0000","last_undegraded":"2023-07-12T15:27:49.954638+0000","last_fullsized":"2023-07-12T15:27:49.954638+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:38:35.484831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956803+0000","last_change":"2023-07-12T15:27:42.135255+0000","last_active":"2023-07-12T15:27:49.956803+0000","last_peered":"2023-07-12T15:27:49.956803+0000","last_clean":"2023-07-12T15:27:49.956803+0000","last_became_active":"2023-07-12T15:27:42.134833+0000","last_became_peered":"2023-07-12T15:27:42.134833+0000","last_unstale":"2023-07-12T15:27:49.956803+0000","last_undegraded":"2023-07-12T15:27:49.956803+0000","last_fullsized":"2023-07-12T15:27:49.956803+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:47:00.249711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166890+0000","last_change":"2023-07-12T15:27:46.156648+0000","last_active":"2023-07-12T15:27:50.166890+0000","last_peered":"2023-07-12T15:27:50.166890+0000","last_clean":"2023-07-12T15:27:50.166890+0000","last_became_active":"2023-07-12T15:27:46.156356+0000","last_became_peered":"2023-07-12T15:27:46.156356+0000","last_unstale":"2023-07-12T15:27:50.166890+0000","last_undegraded":"2023-07-12T15:27:50.166890+0000","last_fullsized":"2023-07-12T15:27:50.166890+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:54:06.704264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'11","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:28.814128+0000","last_change":"2023-07-12T15:27:44.153247+0000","last_active":"2023-07-12T15:28:28.814128+0000","last_peered":"2023-07-12T15:28:28.814128+0000","last_clean":"2023-07-12T15:28:28.814128+0000","last_became_active":"2023-07-12T15:27:44.152959+0000","last_became_peered":"2023-07-12T15:27:44.152959+0000","last_unstale":"2023-07-12T15:28:28.814128+0000","last_undegraded":"2023-07-12T15:28:28.814128+0000","last_fullsized":"2023-07-12T15:28:28.814128+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:33:06.982607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954602+0000","last_change":"2023-07-12T15:27:40.131849+0000","last_active":"2023-07-12T15:27:49.954602+0000","last_peered":"2023-07-12T15:27:49.954602+0000","last_clean":"2023-07-12T15:27:49.954602+0000","last_became_active":"2023-07-12T15:27:40.131588+0000","last_became_peered":"2023-07-12T15:27:40.131588+0000","last_unstale":"2023-07-12T15:27:49.954602+0000","last_undegraded":"2023-07-12T15:27:49.954602+0000","last_fullsized":"2023-07-12T15:27:49.954602+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:00:12.318976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166937+0000","last_change":"2023-07-12T15:27:42.138771+0000","last_active":"2023-07-12T15:27:50.166937+0000","last_peered":"2023-07-12T15:27:50.166937+0000","last_clean":"2023-07-12T15:27:50.166937+0000","last_became_active":"2023-07-12T15:27:42.138522+0000","last_became_peered":"2023-07-12T15:27:42.138522+0000","last_unstale":"2023-07-12T15:27:50.166937+0000","last_undegraded":"2023-07-12T15:27:50.166937+0000","last_fullsized":"2023-07-12T15:27:50.166937+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:22:02.312269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956211+0000","last_change":"2023-07-12T15:27:46.166137+0000","last_active":"2023-07-12T15:27:49.956211+0000","last_peered":"2023-07-12T15:27:49.956211+0000","last_clean":"2023-07-12T15:27:49.956211+0000","last_became_active":"2023-07-12T15:27:46.165854+0000","last_became_peered":"2023-07-12T15:27:46.165854+0000","last_unstale":"2023-07-12T15:27:49.956211+0000","last_undegraded":"2023-07-12T15:27:49.956211+0000","last_fullsized":"2023-07-12T15:27:49.956211+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:26:21.892927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955216+0000","last_change":"2023-07-12T15:27:44.146467+0000","last_active":"2023-07-12T15:27:49.955216+0000","last_peered":"2023-07-12T15:27:49.955216+0000","last_clean":"2023-07-12T15:27:49.955216+0000","last_became_active":"2023-07-12T15:27:44.146136+0000","last_became_peered":"2023-07-12T15:27:44.146136+0000","last_unstale":"2023-07-12T15:27:49.955216+0000","last_undegraded":"2023-07-12T15:27:49.955216+0000","last_fullsized":"2023-07-12T15:27:49.955216+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:39:29.266566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167313+0000","last_change":"2023-07-12T15:27:40.134992+0000","last_active":"2023-07-12T15:27:50.167313+0000","last_peered":"2023-07-12T15:27:50.167313+0000","last_clean":"2023-07-12T15:27:50.167313+0000","last_became_active":"2023-07-12T15:27:40.134695+0000","last_became_peered":"2023-07-12T15:27:40.134695+0000","last_unstale":"2023-07-12T15:27:50.167313+0000","last_undegraded":"2023-07-12T15:27:50.167313+0000","last_fullsized":"2023-07-12T15:27:50.167313+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:26:00.079738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":45,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167058+0000","last_change":"2023-07-12T15:27:42.140726+0000","last_active":"2023-07-12T15:27:50.167058+0000","last_peered":"2023-07-12T15:27:50.167058+0000","last_clean":"2023-07-12T15:27:50.167058+0000","last_became_active":"2023-07-12T15:27:42.140452+0000","last_became_peered":"2023-07-12T15:27:42.140452+0000","last_unstale":"2023-07-12T15:27:50.167058+0000","last_undegraded":"2023-07-12T15:27:50.167058+0000","last_fullsized":"2023-07-12T15:27:50.167058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:55:35.249407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.168086+0000","last_change":"2023-07-12T15:27:46.163623+0000","last_active":"2023-07-12T15:27:50.168086+0000","last_peered":"2023-07-12T15:27:50.168086+0000","last_clean":"2023-07-12T15:27:50.168086+0000","last_became_active":"2023-07-12T15:27:46.163345+0000","last_became_peered":"2023-07-12T15:27:46.163345+0000","last_unstale":"2023-07-12T15:27:50.168086+0000","last_undegraded":"2023-07-12T15:27:50.168086+0000","last_fullsized":"2023-07-12T15:27:50.168086+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:16:38.639243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955948+0000","last_change":"2023-07-12T15:27:44.153904+0000","last_active":"2023-07-12T15:27:49.955948+0000","last_peered":"2023-07-12T15:27:49.955948+0000","last_clean":"2023-07-12T15:27:49.955948+0000","last_became_active":"2023-07-12T15:27:44.153612+0000","last_became_peered":"2023-07-12T15:27:44.153612+0000","last_unstale":"2023-07-12T15:27:49.955948+0000","last_undegraded":"2023-07-12T15:27:49.955948+0000","last_fullsized":"2023-07-12T15:27:49.955948+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:39:37.439894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956103+0000","last_change":"2023-07-12T15:27:40.130234+0000","last_active":"2023-07-12T15:27:49.956103+0000","last_peered":"2023-07-12T15:27:49.956103+0000","last_clean":"2023-07-12T15:27:49.956103+0000","last_became_active":"2023-07-12T15:27:40.129916+0000","last_became_peered":"2023-07-12T15:27:40.129916+0000","last_unstale":"2023-07-12T15:27:49.956103+0000","last_undegraded":"2023-07-12T15:27:49.956103+0000","last_fullsized":"2023-07-12T15:27:49.956103+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:26:58.825905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":48,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166436+0000","last_change":"2023-07-12T15:27:42.132070+0000","last_active":"2023-07-12T15:27:50.166436+0000","last_peered":"2023-07-12T15:27:50.166436+0000","last_clean":"2023-07-12T15:27:50.166436+0000","last_became_active":"2023-07-12T15:27:42.131822+0000","last_became_peered":"2023-07-12T15:27:42.131822+0000","last_unstale":"2023-07-12T15:27:50.166436+0000","last_undegraded":"2023-07-12T15:27:50.166436+0000","last_fullsized":"2023-07-12T15:27:50.166436+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:38:21.874493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165622+0000","last_change":"2023-07-12T15:27:46.169164+0000","last_active":"2023-07-12T15:27:50.165622+0000","last_peered":"2023-07-12T15:27:50.165622+0000","last_clean":"2023-07-12T15:27:50.165622+0000","last_became_active":"2023-07-12T15:27:46.168858+0000","last_became_peered":"2023-07-12T15:27:46.168858+0000","last_unstale":"2023-07-12T15:27:50.165622+0000","last_undegraded":"2023-07-12T15:27:50.165622+0000","last_fullsized":"2023-07-12T15:27:50.165622+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:53:33.591948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955833+0000","last_change":"2023-07-12T15:27:44.149152+0000","last_active":"2023-07-12T15:27:49.955833+0000","last_peered":"2023-07-12T15:27:49.955833+0000","last_clean":"2023-07-12T15:27:49.955833+0000","last_became_active":"2023-07-12T15:27:44.148823+0000","last_became_peered":"2023-07-12T15:27:44.148823+0000","last_unstale":"2023-07-12T15:27:49.955833+0000","last_undegraded":"2023-07-12T15:27:49.955833+0000","last_fullsized":"2023-07-12T15:27:49.955833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:29:02.458613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"60'1","reported_seq":24,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956185+0000","last_change":"2023-07-12T15:27:40.123921+0000","last_active":"2023-07-12T15:27:49.956185+0000","last_peered":"2023-07-12T15:27:49.956185+0000","last_clean":"2023-07-12T15:27:49.956185+0000","last_became_active":"2023-07-12T15:27:40.123639+0000","last_became_peered":"2023-07-12T15:27:40.123639+0000","last_unstale":"2023-07-12T15:27:49.956185+0000","last_undegraded":"2023-07-12T15:27:49.956185+0000","last_fullsized":"2023-07-12T15:27:49.956185+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:59:53.557008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165408+0000","last_change":"2023-07-12T15:27:42.131559+0000","last_active":"2023-07-12T15:27:50.165408+0000","last_peered":"2023-07-12T15:27:50.165408+0000","last_clean":"2023-07-12T15:27:50.165408+0000","last_became_active":"2023-07-12T15:27:42.131301+0000","last_became_peered":"2023-07-12T15:27:42.131301+0000","last_unstale":"2023-07-12T15:27:50.165408+0000","last_undegraded":"2023-07-12T15:27:50.165408+0000","last_fullsized":"2023-07-12T15:27:50.165408+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:34:47.872222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952345+0000","last_change":"2023-07-12T15:27:46.718018+0000","last_active":"2023-07-12T15:27:49.952345+0000","last_peered":"2023-07-12T15:27:49.952345+0000","last_clean":"2023-07-12T15:27:49.952345+0000","last_became_active":"2023-07-12T15:27:46.717731+0000","last_became_peered":"2023-07-12T15:27:46.717731+0000","last_unstale":"2023-07-12T15:27:49.952345+0000","last_undegraded":"2023-07-12T15:27:49.952345+0000","last_fullsized":"2023-07-12T15:27:49.952345+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:24:33.208742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956304+0000","last_change":"2023-07-12T15:27:40.132269+0000","last_active":"2023-07-12T15:27:49.956304+0000","last_peered":"2023-07-12T15:27:49.956304+0000","last_clean":"2023-07-12T15:27:49.956304+0000","last_became_active":"2023-07-12T15:27:40.131890+0000","last_became_peered":"2023-07-12T15:27:40.131890+0000","last_unstale":"2023-07-12T15:27:49.956304+0000","last_undegraded":"2023-07-12T15:27:49.956304+0000","last_fullsized":"2023-07-12T15:27:49.956304+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:50:52.926439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952270+0000","last_change":"2023-07-12T15:27:42.137519+0000","last_active":"2023-07-12T15:27:49.952270+0000","last_peered":"2023-07-12T15:27:49.952270+0000","last_clean":"2023-07-12T15:27:49.952270+0000","last_became_active":"2023-07-12T15:27:42.137252+0000","last_became_peered":"2023-07-12T15:27:42.137252+0000","last_unstale":"2023-07-12T15:27:49.952270+0000","last_undegraded":"2023-07-12T15:27:49.952270+0000","last_fullsized":"2023-07-12T15:27:49.952270+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:52:36.902814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'11","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:28.814392+0000","last_change":"2023-07-12T15:27:44.156440+0000","last_active":"2023-07-12T15:28:28.814392+0000","last_peered":"2023-07-12T15:28:28.814392+0000","last_clean":"2023-07-12T15:28:28.814392+0000","last_became_active":"2023-07-12T15:27:44.156149+0000","last_became_peered":"2023-07-12T15:27:44.156149+0000","last_unstale":"2023-07-12T15:28:28.814392+0000","last_undegraded":"2023-07-12T15:28:28.814392+0000","last_fullsized":"2023-07-12T15:28:28.814392+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:25:54.068870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":44,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1364,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928233999,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28192,"kb_used_data":1240,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716960,"statfs":{"total":95995035648,"available":95966167040,"internally_reserved":0,"allocated":1269760,"data_stored":853900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.53500000000000003}]},{"osd":1,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.432}]},{"osd":2,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.35399999999999998}]},{"osd":3,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.407,"5min":3.407,"15min":3.407},"last":0.48999999999999999}]},{"osd":4,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.54500000000000004}]},{"osd":5,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.48299999999999998}]},{"osd":6,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47099999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495636,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1249280,"data_stored":853333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.435},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.34300000000000003}]},{"osd":1,"last update":"Wed Jul 12 15:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.32000000000000001}]},{"osd":2,"last update":"Wed Jul 12 15:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.628}]},{"osd":3,"last update":"Wed Jul 12 15:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.495}]},{"osd":4,"last update":"Wed Jul 12 15:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.56399999999999995}]},{"osd":5,"last update":"Wed Jul 12 15:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.51400000000000001}]},{"osd":7,"last update":"Wed Jul 12 15:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.61599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27656,"kb_used_data":704,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":720896,"data_stored":265215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34999999999999998,"15min":0.34200000000000003},"min":{"1min":0.23699999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.48399999999999999,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.317},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.38300000000000001,"15min":0.371},"min":{"1min":0.24299999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.33900000000000002}]},{"osd":2,"last update":"Wed Jul 12 15:27:46 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.39100000000000001,"15min":0.39300000000000002},"min":{"1min":0.28499999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.50900000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53500000000000003,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.29699999999999999}]},{"osd":3,"last update":"Wed Jul 12 15:28:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.435},{"interface":"front","average":{"1min":0.436,"5min":0.44,"15min":0.441},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.35099999999999998}]},{"osd":4,"last update":"Wed Jul 12 15:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.44500000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.504},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.51800000000000002}]},{"osd":5,"last update":"Wed Jul 12 15:27:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.49399999999999999}]},{"osd":6,"last update":"Wed Jul 12 15:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.48499999999999999}]},{"osd":7,"last update":"Wed Jul 12 15:28:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.47299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738414,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1273856,"data_stored":854471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jul 12 15:28:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.33400000000000002,"15min":0.32000000000000001},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.33700000000000002},"min":{"1min":0.215,"5min":0.214,"15min":0.214},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.28699999999999998}]},{"osd":2,"last update":"Wed Jul 12 15:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.38700000000000001,"15min":0.38800000000000001},"min":{"1min":0.27900000000000003,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.49399999999999999,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.307},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39400000000000002,"15min":0.39100000000000001},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.33500000000000002}]},{"osd":3,"last update":"Wed Jul 12 15:28:05 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.438,"15min":0.436},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42899999999999999,"15min":0.43099999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.315}]},{"osd":4,"last update":"Wed Jul 12 15:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.65800000000000003}]},{"osd":5,"last update":"Wed Jul 12 15:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.69499999999999995}]},{"osd":6,"last update":"Wed Jul 12 15:28:05 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.55000000000000004}]},{"osd":7,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.55900000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411365,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27596,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717556,"statfs":{"total":95995035648,"available":95966777344,"internally_reserved":0,"allocated":659456,"data_stored":263633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:27:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.314},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.377,"15min":0.375},"min":{"1min":0.311,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.30199999999999999}]},{"osd":1,"last update":"Wed Jul 12 15:27:42 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.39000000000000001,"15min":0.38500000000000001},"min":{"1min":0.253,"5min":0.245,"15min":0.245},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.41499999999999998,"15min":0.41299999999999998},"min":{"1min":0.29599999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.35299999999999998}]},{"osd":3,"last update":"Wed Jul 12 15:28:08 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.47399999999999998,"15min":0.48099999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71799999999999997,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45900000000000002,"15min":0.46700000000000003},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.33700000000000002}]},{"osd":4,"last update":"Wed Jul 12 15:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.65000000000000002,"15min":0.64000000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1110000000000002,"5min":3.1110000000000002,"15min":3.1110000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.66700000000000004,"15min":0.66200000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.6960000000000002,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":0.64700000000000002}]},{"osd":5,"last update":"Wed Jul 12 15:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.68400000000000005}]},{"osd":6,"last update":"Wed Jul 12 15:28:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.70299999999999996}]},{"osd":7,"last update":"Wed Jul 12 15:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":3.0880000000000001,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":2.6349999999999998,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":0.54500000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182433,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27636,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717516,"statfs":{"total":95995035648,"available":95966736384,"internally_reserved":0,"allocated":700416,"data_stored":264993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.39600000000000002,"15min":0.40200000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.55100000000000005,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.50800000000000001}]},{"osd":1,"last update":"Wed Jul 12 15:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53000000000000003,"5min":0.63,"15min":0.63},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.54200000000000004,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.49299999999999999}]},{"osd":2,"last update":"Wed Jul 12 15:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.46400000000000002,"15min":0.46800000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.56699999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.435,"15min":0.44},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.48299999999999998}]},{"osd":4,"last update":"Wed Jul 12 15:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.71399999999999997}]},{"osd":5,"last update":"Wed Jul 12 15:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.72399999999999998}]},{"osd":6,"last update":"Wed Jul 12 15:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.93300000000000005}]},{"osd":7,"last update":"Wed Jul 12 15:28:29 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.878}]}]},{"osd":4,"up_from":30,"seq":128849018909,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27620,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717532,"statfs":{"total":95995035648,"available":95966752768,"internally_reserved":0,"allocated":684032,"data_stored":264041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.57299999999999995,"15min":0.58699999999999997},"min":{"1min":0.35499999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.79300000000000004,"5min":1.268,"15min":1.268},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.39500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80200000000000005,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.49399999999999999}]},{"osd":1,"last update":"Wed Jul 12 15:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.376,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68100000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.504},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.61299999999999999,"15min":0.626},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.80100000000000005,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.67200000000000004}]},{"osd":2,"last update":"Wed Jul 12 15:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.58399999999999996,"15min":0.59499999999999997},"min":{"1min":0.434,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63800000000000001,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Jul 12 15:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.58299999999999996,"15min":0.59499999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85399999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58999999999999997,"15min":0.59799999999999998},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76700000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.56100000000000005}]},{"osd":5,"last update":"Wed Jul 12 15:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.747},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.63800000000000001}]},{"osd":6,"last update":"Wed Jul 12 15:28:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.622},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.77100000000000002}]},{"osd":7,"last update":"Wed Jul 12 15:28:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.65500000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757272,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27600,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":663552,"data_stored":264056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:27:43 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.7149999999999999,"5min":4.7149999999999999,"15min":4.7149999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.53400000000000003}]},{"osd":1,"last update":"Wed Jul 12 15:27:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":4.4710000000000001,"5min":4.4710000000000001,"15min":4.4710000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.7170000000000001,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":0.45900000000000002}]},{"osd":2,"last update":"Wed Jul 12 15:27:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.5659999999999998,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":6.1349999999999998,"5min":6.1349999999999998,"15min":6.1349999999999998},"last":0.55700000000000005}]},{"osd":3,"last update":"Wed Jul 12 15:27:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":3.5859999999999999,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.47999999999999998}]},{"osd":4,"last update":"Wed Jul 12 15:27:43 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.5880000000000001,"5min":2.5880000000000001,"15min":2.5880000000000001},"last":0.499},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.4180000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":0.63}]},{"osd":6,"last update":"Wed Jul 12 15:28:08 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.9459999999999997,"5min":4.9459999999999997,"15min":4.9459999999999997},"last":0.68500000000000005}]},{"osd":7,"last update":"Wed Jul 12 15:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":3.3610000000000002,"5min":3.3610000000000002,"15min":3.3610000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.633,"5min":4.633,"15min":4.633},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-12T15:28:38.603 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph pg dump --format=json 2023-07-12T15:28:38.735 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:38 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[132429]: ts=2023-07-12T15:28:38.479Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000579695s 2023-07-12T15:28:39.157 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:39.676 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:39 smithi069 ceph-mon[102777]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-12T15:28:39.677 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:39 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:39.677 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:39 smithi069 ceph-mon[94241]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-12T15:28:39.677 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:39 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:39.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:39 smithi145 ceph-mon[95801]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-12T15:28:39.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:39 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:28:40.395 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.391+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3701621676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b41054c0 msgr2=0x7f25b410d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:40.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.391+0000 7f25b8cbd700 1 --2- 172.21.15.69:0/3701621676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b41054c0 0x7f25b410d040 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009fc0 tx=0x7f25ac00b330 comp rx=0 tx=0).stop 2023-07-12T15:28:40.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.391+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3701621676 shutdown_connections 2023-07-12T15:28:40.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.391+0000 7f25b8cbd700 1 --2- 172.21.15.69:0/3701621676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b41054c0 0x7f25b410d040 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:40.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.391+0000 7f25b8cbd700 1 --2- 172.21.15.69:0/3701621676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25b4100470 0x7f25b4100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:40.396 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.391+0000 7f25b8cbd700 1 --2- 172.21.15.69:0/3701621676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4100e10 0x7f25b4104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:40.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.391+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3701621676 >> 172.21.15.69:0/3701621676 conn(0x7f25b4078190 msgr2=0x7f25b4078590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:40.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.391+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3701621676 shutdown_connections 2023-07-12T15:28:40.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.391+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3701621676 wait complete. 2023-07-12T15:28:40.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.392+0000 7f25b8cbd700 1 Processor -- start 2023-07-12T15:28:40.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.392+0000 7f25b8cbd700 1 -- start start 2023-07-12T15:28:40.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.392+0000 7f25b8cbd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25b4100470 0x7f25b419b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:40.397 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.392+0000 7f25b8cbd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4100e10 0x7f25b419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:40.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.392+0000 7f25b8cbd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b41054c0 0x7f25b41958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:40.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.392+0000 7f25b8cbd700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b4195f10 con 0x7f25b4100e10 2023-07-12T15:28:40.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.392+0000 7f25b8cbd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b4196080 con 0x7f25b4100470 2023-07-12T15:28:40.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.392+0000 7f25b8cbd700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b4196240 con 0x7f25b41054c0 2023-07-12T15:28:40.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.392+0000 7f25b1d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4100e10 0x7f25b419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:40.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.392+0000 7f25b1d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4100e10 0x7f25b419bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41532/0 (socket says 172.21.15.69:41532) 2023-07-12T15:28:40.398 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.392+0000 7f25b1d9b700 1 -- 172.21.15.69:0/3970400148 learned_addr learned my addr 172.21.15.69:0/3970400148 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:40.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.392+0000 7f25b2d9d700 1 --2- 172.21.15.69:0/3970400148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b41054c0 0x7f25b41958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:40.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.392+0000 7f25b259c700 1 --2- 172.21.15.69:0/3970400148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25b4100470 0x7f25b419b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:40.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.393+0000 7f25b1d9b700 1 -- 172.21.15.69:0/3970400148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b41054c0 msgr2=0x7f25b41958a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:40.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.393+0000 7f25b1d9b700 1 --2- 172.21.15.69:0/3970400148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b41054c0 0x7f25b41958a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:40.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.393+0000 7f25b1d9b700 1 -- 172.21.15.69:0/3970400148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25b4100470 msgr2=0x7f25b419b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:40.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.393+0000 7f25b1d9b700 1 --2- 172.21.15.69:0/3970400148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25b4100470 0x7f25b419b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:40.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.393+0000 7f25b1d9b700 1 -- 172.21.15.69:0/3970400148 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a40095f0 con 0x7f25b4100e10 2023-07-12T15:28:40.399 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.393+0000 7f25b2d9d700 1 --2- 172.21.15.69:0/3970400148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b41054c0 0x7f25b41958a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:28:40.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.393+0000 7f25b1d9b700 1 --2- 172.21.15.69:0/3970400148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4100e10 0x7f25b419bd60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f25a400ec70 tx=0x7f25a400ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:40.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.393+0000 7f25ab7fe700 1 -- 172.21.15.69:0/3970400148 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a40099d0 con 0x7f25b4100e10 2023-07-12T15:28:40.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.393+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3970400148 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac009c40 con 0x7f25b4100e10 2023-07-12T15:28:40.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.393+0000 7f25ab7fe700 1 -- 172.21.15.69:0/3970400148 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f25a40044b0 con 0x7f25b4100e10 2023-07-12T15:28:40.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.393+0000 7f25ab7fe700 1 -- 172.21.15.69:0/3970400148 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a4003e00 con 0x7f25b4100e10 2023-07-12T15:28:40.400 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.393+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3970400148 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b4072f00 con 0x7f25b4100e10 2023-07-12T15:28:40.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.394+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3970400148 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b4196710 con 0x7f25b4100e10 2023-07-12T15:28:40.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.394+0000 7f25ab7fe700 1 -- 172.21.15.69:0/3970400148 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f25a4004000 con 0x7f25b4100e10 2023-07-12T15:28:40.401 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.395+0000 7f25ab7fe700 1 --2- 172.21.15.69:0/3970400148 >> v2:172.21.15.69:6800/1242716914 conn(0x7f25a006c810 0x7f25a006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:40.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.395+0000 7f25ab7fe700 1 -- 172.21.15.69:0/3970400148 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f25a4014070 con 0x7f25b4100e10 2023-07-12T15:28:40.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.395+0000 7f25b259c700 1 --2- 172.21.15.69:0/3970400148 >> v2:172.21.15.69:6800/1242716914 conn(0x7f25a006c810 0x7f25a006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:40.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.395+0000 7f25b259c700 1 --2- 172.21.15.69:0/3970400148 >> v2:172.21.15.69:6800/1242716914 conn(0x7f25a006c810 0x7f25a006ecc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f259c005950 tx=0x7f259c00b410 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:40.402 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.399+0000 7f25ab7fe700 1 -- 172.21.15.69:0/3970400148 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f25b4196710 con 0x7f25b4100e10 2023-07-12T15:28:40.586 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.583+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3970400148 --> v2:172.21.15.69:6800/1242716914 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f25b4196710 con 0x7f25a006c810 2023-07-12T15:28:40.596 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.594+0000 7f25ab7fe700 1 -- 172.21.15.69:0/3970400148 <== mgr.24524 v2:172.21.15.69:6800/1242716914 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+384158 (secure 0 0 0) 0x7f25b4196710 con 0x7f25a006c810 2023-07-12T15:28:40.600 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:28:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.597+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3970400148 >> v2:172.21.15.69:6800/1242716914 conn(0x7f25a006c810 msgr2=0x7f25a006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.597+0000 7f25b8cbd700 1 --2- 172.21.15.69:0/3970400148 >> v2:172.21.15.69:6800/1242716914 conn(0x7f25a006c810 0x7f25a006ecc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f259c005950 tx=0x7f259c00b410 comp rx=0 tx=0).stop 2023-07-12T15:28:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.597+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3970400148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4100e10 msgr2=0x7f25b419bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.597+0000 7f25b8cbd700 1 --2- 172.21.15.69:0/3970400148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4100e10 0x7f25b419bd60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f25a400ec70 tx=0x7f25a400ef80 comp rx=0 tx=0).stop 2023-07-12T15:28:40.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.597+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3970400148 shutdown_connections 2023-07-12T15:28:40.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.597+0000 7f25b8cbd700 1 --2- 172.21.15.69:0/3970400148 >> v2:172.21.15.69:6800/1242716914 conn(0x7f25a006c810 0x7f25a006ecc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:40.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.597+0000 7f25b8cbd700 1 --2- 172.21.15.69:0/3970400148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b41054c0 0x7f25b41958a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:40.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.597+0000 7f25b8cbd700 1 --2- 172.21.15.69:0/3970400148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25b4100470 0x7f25b419b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:40.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.597+0000 7f25b8cbd700 1 --2- 172.21.15.69:0/3970400148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4100e10 0x7f25b419bd60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:40.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.597+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3970400148 >> 172.21.15.69:0/3970400148 conn(0x7f25b4078190 msgr2=0x7f25b40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:40.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.597+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3970400148 shutdown_connections 2023-07-12T15:28:40.603 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:40.598+0000 7f25b8cbd700 1 -- 172.21.15.69:0/3970400148 wait complete. 2023-07-12T15:28:40.604 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-07-12T15:28:40.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:40 smithi069 ceph-mon[102777]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:40.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:40 smithi069 ceph-mon[94241]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:40.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:40 smithi145 ceph-mon[95801]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:41.010 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-07-12T15:28:39.372504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596224,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":986,"num_read_kb":832,"num_write":609,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":622,"ondisk_log_size":622,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":222668,"kb_used_data":7052,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":749738548,"statfs":{"total":767960285184,"available":767732273152,"internally_reserved":0,"allocated":7221248,"data_stored":3883642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002439"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166198+0000","last_change":"2023-07-12T15:27:46.719957+0000","last_active":"2023-07-12T15:27:50.166198+0000","last_peered":"2023-07-12T15:27:50.166198+0000","last_clean":"2023-07-12T15:27:50.166198+0000","last_became_active":"2023-07-12T15:27:46.719622+0000","last_became_peered":"2023-07-12T15:27:46.719622+0000","last_unstale":"2023-07-12T15:27:50.166198+0000","last_undegraded":"2023-07-12T15:27:50.166198+0000","last_fullsized":"2023-07-12T15:27:50.166198+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:34:02.770908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166029+0000","last_change":"2023-07-12T15:27:44.145601+0000","last_active":"2023-07-12T15:27:50.166029+0000","last_peered":"2023-07-12T15:27:50.166029+0000","last_clean":"2023-07-12T15:27:50.166029+0000","last_became_active":"2023-07-12T15:27:44.145284+0000","last_became_peered":"2023-07-12T15:27:44.145284+0000","last_unstale":"2023-07-12T15:27:50.166029+0000","last_undegraded":"2023-07-12T15:27:50.166029+0000","last_fullsized":"2023-07-12T15:27:50.166029+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:01:19.271063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166333+0000","last_change":"2023-07-12T15:27:40.130960+0000","last_active":"2023-07-12T15:27:50.166333+0000","last_peered":"2023-07-12T15:27:50.166333+0000","last_clean":"2023-07-12T15:27:50.166333+0000","last_became_active":"2023-07-12T15:27:40.130721+0000","last_became_peered":"2023-07-12T15:27:40.130721+0000","last_unstale":"2023-07-12T15:27:50.166333+0000","last_undegraded":"2023-07-12T15:27:50.166333+0000","last_fullsized":"2023-07-12T15:27:50.166333+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:22:03.109709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165359+0000","last_change":"2023-07-12T15:27:42.134026+0000","last_active":"2023-07-12T15:27:50.165359+0000","last_peered":"2023-07-12T15:27:50.165359+0000","last_clean":"2023-07-12T15:27:50.165359+0000","last_became_active":"2023-07-12T15:27:42.133731+0000","last_became_peered":"2023-07-12T15:27:42.133731+0000","last_unstale":"2023-07-12T15:27:50.165359+0000","last_undegraded":"2023-07-12T15:27:50.165359+0000","last_fullsized":"2023-07-12T15:27:50.165359+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:23:38.384520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166460+0000","last_change":"2023-07-12T15:27:46.163608+0000","last_active":"2023-07-12T15:27:50.166460+0000","last_peered":"2023-07-12T15:27:50.166460+0000","last_clean":"2023-07-12T15:27:50.166460+0000","last_became_active":"2023-07-12T15:27:46.163185+0000","last_became_peered":"2023-07-12T15:27:46.163185+0000","last_unstale":"2023-07-12T15:27:50.166460+0000","last_undegraded":"2023-07-12T15:27:50.166460+0000","last_fullsized":"2023-07-12T15:27:50.166460+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:51:17.234063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954488+0000","last_change":"2023-07-12T15:27:44.148220+0000","last_active":"2023-07-12T15:27:49.954488+0000","last_peered":"2023-07-12T15:27:49.954488+0000","last_clean":"2023-07-12T15:27:49.954488+0000","last_became_active":"2023-07-12T15:27:44.147914+0000","last_became_peered":"2023-07-12T15:27:44.147914+0000","last_unstale":"2023-07-12T15:27:49.954488+0000","last_undegraded":"2023-07-12T15:27:49.954488+0000","last_fullsized":"2023-07-12T15:27:49.954488+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:33:22.313311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"60'1","reported_seq":25,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166799+0000","last_change":"2023-07-12T15:27:40.129257+0000","last_active":"2023-07-12T15:27:50.166799+0000","last_peered":"2023-07-12T15:27:50.166799+0000","last_clean":"2023-07-12T15:27:50.166799+0000","last_became_active":"2023-07-12T15:27:40.128960+0000","last_became_peered":"2023-07-12T15:27:40.128960+0000","last_unstale":"2023-07-12T15:27:50.166799+0000","last_undegraded":"2023-07-12T15:27:50.166799+0000","last_fullsized":"2023-07-12T15:27:50.166799+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:36:36.864844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165931+0000","last_change":"2023-07-12T15:27:42.135394+0000","last_active":"2023-07-12T15:27:50.165931+0000","last_peered":"2023-07-12T15:27:50.165931+0000","last_clean":"2023-07-12T15:27:50.165931+0000","last_became_active":"2023-07-12T15:27:42.134943+0000","last_became_peered":"2023-07-12T15:27:42.134943+0000","last_unstale":"2023-07-12T15:27:50.165931+0000","last_undegraded":"2023-07-12T15:27:50.165931+0000","last_fullsized":"2023-07-12T15:27:50.165931+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:14:16.773828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952302+0000","last_change":"2023-07-12T15:27:46.164091+0000","last_active":"2023-07-12T15:27:49.952302+0000","last_peered":"2023-07-12T15:27:49.952302+0000","last_clean":"2023-07-12T15:27:49.952302+0000","last_became_active":"2023-07-12T15:27:46.163796+0000","last_became_peered":"2023-07-12T15:27:46.163796+0000","last_unstale":"2023-07-12T15:27:49.952302+0000","last_undegraded":"2023-07-12T15:27:49.952302+0000","last_fullsized":"2023-07-12T15:27:49.952302+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T20:54:16.307133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166124+0000","last_change":"2023-07-12T15:27:44.155072+0000","last_active":"2023-07-12T15:27:50.166124+0000","last_peered":"2023-07-12T15:27:50.166124+0000","last_clean":"2023-07-12T15:27:50.166124+0000","last_became_active":"2023-07-12T15:27:44.154743+0000","last_became_peered":"2023-07-12T15:27:44.154743+0000","last_unstale":"2023-07-12T15:27:50.166124+0000","last_undegraded":"2023-07-12T15:27:50.166124+0000","last_fullsized":"2023-07-12T15:27:50.166124+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:52:24.294886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167786+0000","last_change":"2023-07-12T15:27:40.134604+0000","last_active":"2023-07-12T15:27:50.167786+0000","last_peered":"2023-07-12T15:27:50.167786+0000","last_clean":"2023-07-12T15:27:50.167786+0000","last_became_active":"2023-07-12T15:27:40.134306+0000","last_became_peered":"2023-07-12T15:27:40.134306+0000","last_unstale":"2023-07-12T15:27:50.167786+0000","last_undegraded":"2023-07-12T15:27:50.167786+0000","last_fullsized":"2023-07-12T15:27:50.167786+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:24:48.235457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":27,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166635+0000","last_change":"2023-07-12T15:27:42.138719+0000","last_active":"2023-07-12T15:27:50.166635+0000","last_peered":"2023-07-12T15:27:50.166635+0000","last_clean":"2023-07-12T15:27:50.166635+0000","last_became_active":"2023-07-12T15:27:42.138441+0000","last_became_peered":"2023-07-12T15:27:42.138441+0000","last_unstale":"2023-07-12T15:27:50.166635+0000","last_undegraded":"2023-07-12T15:27:50.166635+0000","last_fullsized":"2023-07-12T15:27:50.166635+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:40:34.302700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"67'1","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166957+0000","last_change":"2023-07-12T15:27:46.169583+0000","last_active":"2023-07-12T15:27:50.166957+0000","last_peered":"2023-07-12T15:27:50.166957+0000","last_clean":"2023-07-12T15:27:50.166957+0000","last_became_active":"2023-07-12T15:27:46.169216+0000","last_became_peered":"2023-07-12T15:27:46.169216+0000","last_unstale":"2023-07-12T15:27:50.166957+0000","last_undegraded":"2023-07-12T15:27:50.166957+0000","last_fullsized":"2023-07-12T15:27:50.166957+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:11:59.801697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952583+0000","last_change":"2023-07-12T15:27:44.156992+0000","last_active":"2023-07-12T15:27:49.952583+0000","last_peered":"2023-07-12T15:27:49.952583+0000","last_clean":"2023-07-12T15:27:49.952583+0000","last_became_active":"2023-07-12T15:27:44.156668+0000","last_became_peered":"2023-07-12T15:27:44.156668+0000","last_unstale":"2023-07-12T15:27:49.952583+0000","last_undegraded":"2023-07-12T15:27:49.952583+0000","last_fullsized":"2023-07-12T15:27:49.952583+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:41:39.142746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165923+0000","last_change":"2023-07-12T15:27:40.133664+0000","last_active":"2023-07-12T15:27:50.165923+0000","last_peered":"2023-07-12T15:27:50.165923+0000","last_clean":"2023-07-12T15:27:50.165923+0000","last_became_active":"2023-07-12T15:27:40.131966+0000","last_became_peered":"2023-07-12T15:27:40.131966+0000","last_unstale":"2023-07-12T15:27:50.165923+0000","last_undegraded":"2023-07-12T15:27:50.165923+0000","last_fullsized":"2023-07-12T15:27:50.165923+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:37:27.700710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166539+0000","last_change":"2023-07-12T15:27:42.135430+0000","last_active":"2023-07-12T15:27:50.166539+0000","last_peered":"2023-07-12T15:27:50.166539+0000","last_clean":"2023-07-12T15:27:50.166539+0000","last_became_active":"2023-07-12T15:27:42.135174+0000","last_became_peered":"2023-07-12T15:27:42.135174+0000","last_unstale":"2023-07-12T15:27:50.166539+0000","last_undegraded":"2023-07-12T15:27:50.166539+0000","last_fullsized":"2023-07-12T15:27:50.166539+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:22:43.754172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166903+0000","last_change":"2023-07-12T15:27:46.717310+0000","last_active":"2023-07-12T15:27:50.166903+0000","last_peered":"2023-07-12T15:27:50.166903+0000","last_clean":"2023-07-12T15:27:50.166903+0000","last_became_active":"2023-07-12T15:27:46.716925+0000","last_became_peered":"2023-07-12T15:27:46.716925+0000","last_unstale":"2023-07-12T15:27:50.166903+0000","last_undegraded":"2023-07-12T15:27:50.166903+0000","last_fullsized":"2023-07-12T15:27:50.166903+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:33:10.467570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167066+0000","last_change":"2023-07-12T15:27:44.149841+0000","last_active":"2023-07-12T15:27:50.167066+0000","last_peered":"2023-07-12T15:27:50.167066+0000","last_clean":"2023-07-12T15:27:50.167066+0000","last_became_active":"2023-07-12T15:27:44.149447+0000","last_became_peered":"2023-07-12T15:27:44.149447+0000","last_unstale":"2023-07-12T15:27:50.167066+0000","last_undegraded":"2023-07-12T15:27:50.167066+0000","last_fullsized":"2023-07-12T15:27:50.167066+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:21:42.788690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167404+0000","last_change":"2023-07-12T15:27:40.130768+0000","last_active":"2023-07-12T15:27:50.167404+0000","last_peered":"2023-07-12T15:27:50.167404+0000","last_clean":"2023-07-12T15:27:50.167404+0000","last_became_active":"2023-07-12T15:27:40.130515+0000","last_became_peered":"2023-07-12T15:27:40.130515+0000","last_unstale":"2023-07-12T15:27:50.167404+0000","last_undegraded":"2023-07-12T15:27:50.167404+0000","last_fullsized":"2023-07-12T15:27:50.167404+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T15:29:43.002441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":40,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165470+0000","last_change":"2023-07-12T15:27:42.137841+0000","last_active":"2023-07-12T15:27:50.165470+0000","last_peered":"2023-07-12T15:27:50.165470+0000","last_clean":"2023-07-12T15:27:50.165470+0000","last_became_active":"2023-07-12T15:27:42.137587+0000","last_became_peered":"2023-07-12T15:27:42.137587+0000","last_unstale":"2023-07-12T15:27:50.165470+0000","last_undegraded":"2023-07-12T15:27:50.165470+0000","last_fullsized":"2023-07-12T15:27:50.165470+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:40:54.563480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167000+0000","last_change":"2023-07-12T15:27:46.717815+0000","last_active":"2023-07-12T15:27:50.167000+0000","last_peered":"2023-07-12T15:27:50.167000+0000","last_clean":"2023-07-12T15:27:50.167000+0000","last_became_active":"2023-07-12T15:27:46.717538+0000","last_became_peered":"2023-07-12T15:27:46.717538+0000","last_unstale":"2023-07-12T15:27:50.167000+0000","last_undegraded":"2023-07-12T15:27:50.167000+0000","last_fullsized":"2023-07-12T15:27:50.167000+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:31:05.594988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956314+0000","last_change":"2023-07-12T15:27:44.145926+0000","last_active":"2023-07-12T15:27:49.956314+0000","last_peered":"2023-07-12T15:27:49.956314+0000","last_clean":"2023-07-12T15:27:49.956314+0000","last_became_active":"2023-07-12T15:27:44.145632+0000","last_became_peered":"2023-07-12T15:27:44.145632+0000","last_unstale":"2023-07-12T15:27:49.956314+0000","last_undegraded":"2023-07-12T15:27:49.956314+0000","last_fullsized":"2023-07-12T15:27:49.956314+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:55:14.743990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166632+0000","last_change":"2023-07-12T15:27:40.133269+0000","last_active":"2023-07-12T15:27:50.166632+0000","last_peered":"2023-07-12T15:27:50.166632+0000","last_clean":"2023-07-12T15:27:50.166632+0000","last_became_active":"2023-07-12T15:27:40.132972+0000","last_became_peered":"2023-07-12T15:27:40.132972+0000","last_unstale":"2023-07-12T15:27:50.166632+0000","last_undegraded":"2023-07-12T15:27:50.166632+0000","last_fullsized":"2023-07-12T15:27:50.166632+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:40:47.394603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956678+0000","last_change":"2023-07-12T15:27:42.133950+0000","last_active":"2023-07-12T15:27:49.956678+0000","last_peered":"2023-07-12T15:27:49.956678+0000","last_clean":"2023-07-12T15:27:49.956678+0000","last_became_active":"2023-07-12T15:27:42.133660+0000","last_became_peered":"2023-07-12T15:27:42.133660+0000","last_unstale":"2023-07-12T15:27:49.956678+0000","last_undegraded":"2023-07-12T15:27:49.956678+0000","last_fullsized":"2023-07-12T15:27:49.956678+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:20:03.074678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955396+0000","last_change":"2023-07-12T15:27:46.160478+0000","last_active":"2023-07-12T15:27:49.955396+0000","last_peered":"2023-07-12T15:27:49.955396+0000","last_clean":"2023-07-12T15:27:49.955396+0000","last_became_active":"2023-07-12T15:27:46.160198+0000","last_became_peered":"2023-07-12T15:27:46.160198+0000","last_unstale":"2023-07-12T15:27:49.955396+0000","last_undegraded":"2023-07-12T15:27:49.955396+0000","last_fullsized":"2023-07-12T15:27:49.955396+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T20:48:27.514758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166821+0000","last_change":"2023-07-12T15:27:44.150054+0000","last_active":"2023-07-12T15:27:50.166821+0000","last_peered":"2023-07-12T15:27:50.166821+0000","last_clean":"2023-07-12T15:27:50.166821+0000","last_became_active":"2023-07-12T15:27:44.149751+0000","last_became_peered":"2023-07-12T15:27:44.149751+0000","last_unstale":"2023-07-12T15:27:50.166821+0000","last_undegraded":"2023-07-12T15:27:50.166821+0000","last_fullsized":"2023-07-12T15:27:50.166821+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:58:42.614488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167064+0000","last_change":"2023-07-12T15:27:40.128889+0000","last_active":"2023-07-12T15:27:50.167064+0000","last_peered":"2023-07-12T15:27:50.167064+0000","last_clean":"2023-07-12T15:27:50.167064+0000","last_became_active":"2023-07-12T15:27:40.128651+0000","last_became_peered":"2023-07-12T15:27:40.128651+0000","last_unstale":"2023-07-12T15:27:50.167064+0000","last_undegraded":"2023-07-12T15:27:50.167064+0000","last_fullsized":"2023-07-12T15:27:50.167064+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:15:47.395585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166871+0000","last_change":"2023-07-12T15:27:42.131973+0000","last_active":"2023-07-12T15:27:50.166871+0000","last_peered":"2023-07-12T15:27:50.166871+0000","last_clean":"2023-07-12T15:27:50.166871+0000","last_became_active":"2023-07-12T15:27:42.131669+0000","last_became_peered":"2023-07-12T15:27:42.131669+0000","last_unstale":"2023-07-12T15:27:50.166871+0000","last_undegraded":"2023-07-12T15:27:50.166871+0000","last_fullsized":"2023-07-12T15:27:50.166871+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:46:42.718907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166252+0000","last_change":"2023-07-12T15:27:46.167326+0000","last_active":"2023-07-12T15:27:50.166252+0000","last_peered":"2023-07-12T15:27:50.166252+0000","last_clean":"2023-07-12T15:27:50.166252+0000","last_became_active":"2023-07-12T15:27:46.167052+0000","last_became_peered":"2023-07-12T15:27:46.167052+0000","last_unstale":"2023-07-12T15:27:50.166252+0000","last_undegraded":"2023-07-12T15:27:50.166252+0000","last_fullsized":"2023-07-12T15:27:50.166252+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:10:15.773772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'11","reported_seq":43,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:28.814440+0000","last_change":"2023-07-12T15:27:44.152180+0000","last_active":"2023-07-12T15:28:28.814440+0000","last_peered":"2023-07-12T15:28:28.814440+0000","last_clean":"2023-07-12T15:28:28.814440+0000","last_became_active":"2023-07-12T15:27:44.151874+0000","last_became_peered":"2023-07-12T15:27:44.151874+0000","last_unstale":"2023-07-12T15:28:28.814440+0000","last_undegraded":"2023-07-12T15:28:28.814440+0000","last_fullsized":"2023-07-12T15:28:28.814440+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:58:02.492468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952222+0000","last_change":"2023-07-12T15:27:40.134571+0000","last_active":"2023-07-12T15:27:49.952222+0000","last_peered":"2023-07-12T15:27:49.952222+0000","last_clean":"2023-07-12T15:27:49.952222+0000","last_became_active":"2023-07-12T15:27:40.134273+0000","last_became_peered":"2023-07-12T15:27:40.134273+0000","last_unstale":"2023-07-12T15:27:49.952222+0000","last_undegraded":"2023-07-12T15:27:49.952222+0000","last_fullsized":"2023-07-12T15:27:49.952222+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:36:11.998289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166739+0000","last_change":"2023-07-12T15:27:42.136459+0000","last_active":"2023-07-12T15:27:50.166739+0000","last_peered":"2023-07-12T15:27:50.166739+0000","last_clean":"2023-07-12T15:27:50.166739+0000","last_became_active":"2023-07-12T15:27:42.136026+0000","last_became_peered":"2023-07-12T15:27:42.136026+0000","last_unstale":"2023-07-12T15:27:50.166739+0000","last_undegraded":"2023-07-12T15:27:50.166739+0000","last_fullsized":"2023-07-12T15:27:50.166739+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:03:36.934879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167358+0000","last_change":"2023-07-12T15:27:46.717953+0000","last_active":"2023-07-12T15:27:50.167358+0000","last_peered":"2023-07-12T15:27:50.167358+0000","last_clean":"2023-07-12T15:27:50.167358+0000","last_became_active":"2023-07-12T15:27:46.717380+0000","last_became_peered":"2023-07-12T15:27:46.717380+0000","last_unstale":"2023-07-12T15:27:50.167358+0000","last_undegraded":"2023-07-12T15:27:50.167358+0000","last_fullsized":"2023-07-12T15:27:50.167358+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:19:21.542061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.168343+0000","last_change":"2023-07-12T15:27:44.155961+0000","last_active":"2023-07-12T15:27:50.168343+0000","last_peered":"2023-07-12T15:27:50.168343+0000","last_clean":"2023-07-12T15:27:50.168343+0000","last_became_active":"2023-07-12T15:27:44.155640+0000","last_became_peered":"2023-07-12T15:27:44.155640+0000","last_unstale":"2023-07-12T15:27:50.168343+0000","last_undegraded":"2023-07-12T15:27:50.168343+0000","last_fullsized":"2023-07-12T15:27:50.168343+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:18:07.314044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167189+0000","last_change":"2023-07-12T15:27:40.130730+0000","last_active":"2023-07-12T15:27:50.167189+0000","last_peered":"2023-07-12T15:27:50.167189+0000","last_clean":"2023-07-12T15:27:50.167189+0000","last_became_active":"2023-07-12T15:27:40.130412+0000","last_became_peered":"2023-07-12T15:27:40.130412+0000","last_unstale":"2023-07-12T15:27:50.167189+0000","last_undegraded":"2023-07-12T15:27:50.167189+0000","last_fullsized":"2023-07-12T15:27:50.167189+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:36:04.948411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167413+0000","last_change":"2023-07-12T15:27:42.132862+0000","last_active":"2023-07-12T15:27:50.167413+0000","last_peered":"2023-07-12T15:27:50.167413+0000","last_clean":"2023-07-12T15:27:50.167413+0000","last_became_active":"2023-07-12T15:27:42.132592+0000","last_became_peered":"2023-07-12T15:27:42.132592+0000","last_unstale":"2023-07-12T15:27:50.167413+0000","last_undegraded":"2023-07-12T15:27:50.167413+0000","last_fullsized":"2023-07-12T15:27:50.167413+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:36:19.909414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165953+0000","last_change":"2023-07-12T15:27:46.167372+0000","last_active":"2023-07-12T15:27:50.165953+0000","last_peered":"2023-07-12T15:27:50.165953+0000","last_clean":"2023-07-12T15:27:50.165953+0000","last_became_active":"2023-07-12T15:27:46.167024+0000","last_became_peered":"2023-07-12T15:27:46.167024+0000","last_unstale":"2023-07-12T15:27:50.165953+0000","last_undegraded":"2023-07-12T15:27:50.165953+0000","last_fullsized":"2023-07-12T15:27:50.165953+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:40:36.392499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167636+0000","last_change":"2023-07-12T15:27:44.158812+0000","last_active":"2023-07-12T15:27:50.167636+0000","last_peered":"2023-07-12T15:27:50.167636+0000","last_clean":"2023-07-12T15:27:50.167636+0000","last_became_active":"2023-07-12T15:27:44.158484+0000","last_became_peered":"2023-07-12T15:27:44.158484+0000","last_unstale":"2023-07-12T15:27:50.167636+0000","last_undegraded":"2023-07-12T15:27:50.167636+0000","last_fullsized":"2023-07-12T15:27:50.167636+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T20:30:56.007120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952742+0000","last_change":"2023-07-12T15:27:40.133310+0000","last_active":"2023-07-12T15:27:49.952742+0000","last_peered":"2023-07-12T15:27:49.952742+0000","last_clean":"2023-07-12T15:27:49.952742+0000","last_became_active":"2023-07-12T15:27:40.132983+0000","last_became_peered":"2023-07-12T15:27:40.132983+0000","last_unstale":"2023-07-12T15:27:49.952742+0000","last_undegraded":"2023-07-12T15:27:49.952742+0000","last_fullsized":"2023-07-12T15:27:49.952742+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:19:34.869750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":23,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165332+0000","last_change":"2023-07-12T15:27:42.136712+0000","last_active":"2023-07-12T15:27:50.165332+0000","last_peered":"2023-07-12T15:27:50.165332+0000","last_clean":"2023-07-12T15:27:50.165332+0000","last_became_active":"2023-07-12T15:27:42.136414+0000","last_became_peered":"2023-07-12T15:27:42.136414+0000","last_unstale":"2023-07-12T15:27:50.165332+0000","last_undegraded":"2023-07-12T15:27:50.165332+0000","last_fullsized":"2023-07-12T15:27:50.165332+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:35:44.585898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166813+0000","last_change":"2023-07-12T15:27:46.167443+0000","last_active":"2023-07-12T15:27:50.166813+0000","last_peered":"2023-07-12T15:27:50.166813+0000","last_clean":"2023-07-12T15:27:50.166813+0000","last_became_active":"2023-07-12T15:27:46.167105+0000","last_became_peered":"2023-07-12T15:27:46.167105+0000","last_unstale":"2023-07-12T15:27:50.166813+0000","last_undegraded":"2023-07-12T15:27:50.166813+0000","last_fullsized":"2023-07-12T15:27:50.166813+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:29:07.739097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954765+0000","last_change":"2023-07-12T15:27:44.154978+0000","last_active":"2023-07-12T15:27:49.954765+0000","last_peered":"2023-07-12T15:27:49.954765+0000","last_clean":"2023-07-12T15:27:49.954765+0000","last_became_active":"2023-07-12T15:27:44.154696+0000","last_became_peered":"2023-07-12T15:27:44.154696+0000","last_unstale":"2023-07-12T15:27:49.954765+0000","last_undegraded":"2023-07-12T15:27:49.954765+0000","last_fullsized":"2023-07-12T15:27:49.954765+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:18:44.337926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956025+0000","last_change":"2023-07-12T15:27:40.128722+0000","last_active":"2023-07-12T15:27:49.956025+0000","last_peered":"2023-07-12T15:27:49.956025+0000","last_clean":"2023-07-12T15:27:49.956025+0000","last_became_active":"2023-07-12T15:27:40.128430+0000","last_became_peered":"2023-07-12T15:27:40.128430+0000","last_unstale":"2023-07-12T15:27:49.956025+0000","last_undegraded":"2023-07-12T15:27:49.956025+0000","last_fullsized":"2023-07-12T15:27:49.956025+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:25:48.743883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166575+0000","last_change":"2023-07-12T15:27:42.135381+0000","last_active":"2023-07-12T15:27:50.166575+0000","last_peered":"2023-07-12T15:27:50.166575+0000","last_clean":"2023-07-12T15:27:50.166575+0000","last_became_active":"2023-07-12T15:27:42.135032+0000","last_became_peered":"2023-07-12T15:27:42.135032+0000","last_unstale":"2023-07-12T15:27:50.166575+0000","last_undegraded":"2023-07-12T15:27:50.166575+0000","last_fullsized":"2023-07-12T15:27:50.166575+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:09:24.118737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166336+0000","last_change":"2023-07-12T15:27:46.169664+0000","last_active":"2023-07-12T15:27:50.166336+0000","last_peered":"2023-07-12T15:27:50.166336+0000","last_clean":"2023-07-12T15:27:50.166336+0000","last_became_active":"2023-07-12T15:27:46.169282+0000","last_became_peered":"2023-07-12T15:27:46.169282+0000","last_unstale":"2023-07-12T15:27:50.166336+0000","last_undegraded":"2023-07-12T15:27:50.166336+0000","last_fullsized":"2023-07-12T15:27:50.166336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:06:20.633426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167179+0000","last_change":"2023-07-12T15:27:44.152244+0000","last_active":"2023-07-12T15:27:50.167179+0000","last_peered":"2023-07-12T15:27:50.167179+0000","last_clean":"2023-07-12T15:27:50.167179+0000","last_became_active":"2023-07-12T15:27:44.151851+0000","last_became_peered":"2023-07-12T15:27:44.151851+0000","last_unstale":"2023-07-12T15:27:50.167179+0000","last_undegraded":"2023-07-12T15:27:50.167179+0000","last_fullsized":"2023-07-12T15:27:50.167179+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:32:30.762564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166212+0000","last_change":"2023-07-12T15:27:40.130987+0000","last_active":"2023-07-12T15:27:50.166212+0000","last_peered":"2023-07-12T15:27:50.166212+0000","last_clean":"2023-07-12T15:27:50.166212+0000","last_became_active":"2023-07-12T15:27:40.130599+0000","last_became_peered":"2023-07-12T15:27:40.130599+0000","last_unstale":"2023-07-12T15:27:50.166212+0000","last_undegraded":"2023-07-12T15:27:50.166212+0000","last_fullsized":"2023-07-12T15:27:50.166212+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T15:28:30.296076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954687+0000","last_change":"2023-07-12T15:27:42.128982+0000","last_active":"2023-07-12T15:27:49.954687+0000","last_peered":"2023-07-12T15:27:49.954687+0000","last_clean":"2023-07-12T15:27:49.954687+0000","last_became_active":"2023-07-12T15:27:42.128687+0000","last_became_peered":"2023-07-12T15:27:42.128687+0000","last_unstale":"2023-07-12T15:27:49.954687+0000","last_undegraded":"2023-07-12T15:27:49.954687+0000","last_fullsized":"2023-07-12T15:27:49.954687+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:16:49.488081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.168258+0000","last_change":"2023-07-12T15:27:46.161337+0000","last_active":"2023-07-12T15:27:50.168258+0000","last_peered":"2023-07-12T15:27:50.168258+0000","last_clean":"2023-07-12T15:27:50.168258+0000","last_became_active":"2023-07-12T15:27:46.161070+0000","last_became_peered":"2023-07-12T15:27:46.161070+0000","last_unstale":"2023-07-12T15:27:50.168258+0000","last_undegraded":"2023-07-12T15:27:50.168258+0000","last_fullsized":"2023-07-12T15:27:50.168258+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:20:31.063627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'11","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:33.814440+0000","last_change":"2023-07-12T15:27:44.151955+0000","last_active":"2023-07-12T15:28:33.814440+0000","last_peered":"2023-07-12T15:28:33.814440+0000","last_clean":"2023-07-12T15:28:33.814440+0000","last_became_active":"2023-07-12T15:27:44.151546+0000","last_became_peered":"2023-07-12T15:27:44.151546+0000","last_unstale":"2023-07-12T15:28:33.814440+0000","last_undegraded":"2023-07-12T15:28:33.814440+0000","last_fullsized":"2023-07-12T15:28:33.814440+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:04:19.255266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165972+0000","last_change":"2023-07-12T15:27:40.122947+0000","last_active":"2023-07-12T15:27:50.165972+0000","last_peered":"2023-07-12T15:27:50.165972+0000","last_clean":"2023-07-12T15:27:50.165972+0000","last_became_active":"2023-07-12T15:27:40.122685+0000","last_became_peered":"2023-07-12T15:27:40.122685+0000","last_unstale":"2023-07-12T15:27:50.165972+0000","last_undegraded":"2023-07-12T15:27:50.165972+0000","last_fullsized":"2023-07-12T15:27:50.165972+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:21:21.256101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952832+0000","last_change":"2023-07-12T15:27:42.131173+0000","last_active":"2023-07-12T15:27:49.952832+0000","last_peered":"2023-07-12T15:27:49.952832+0000","last_clean":"2023-07-12T15:27:49.952832+0000","last_became_active":"2023-07-12T15:27:42.130880+0000","last_became_peered":"2023-07-12T15:27:42.130880+0000","last_unstale":"2023-07-12T15:27:49.952832+0000","last_undegraded":"2023-07-12T15:27:49.952832+0000","last_fullsized":"2023-07-12T15:27:49.952832+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:06:14.663877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166387+0000","last_change":"2023-07-12T15:27:46.168899+0000","last_active":"2023-07-12T15:27:50.166387+0000","last_peered":"2023-07-12T15:27:50.166387+0000","last_clean":"2023-07-12T15:27:50.166387+0000","last_became_active":"2023-07-12T15:27:46.168511+0000","last_became_peered":"2023-07-12T15:27:46.168511+0000","last_unstale":"2023-07-12T15:27:50.166387+0000","last_undegraded":"2023-07-12T15:27:50.166387+0000","last_fullsized":"2023-07-12T15:27:50.166387+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:28:38.893034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'11","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:28.814394+0000","last_change":"2023-07-12T15:27:44.147818+0000","last_active":"2023-07-12T15:28:28.814394+0000","last_peered":"2023-07-12T15:28:28.814394+0000","last_clean":"2023-07-12T15:28:28.814394+0000","last_became_active":"2023-07-12T15:27:44.147589+0000","last_became_peered":"2023-07-12T15:27:44.147589+0000","last_unstale":"2023-07-12T15:28:28.814394+0000","last_undegraded":"2023-07-12T15:28:28.814394+0000","last_fullsized":"2023-07-12T15:28:28.814394+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:56:50.718828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952389+0000","last_change":"2023-07-12T15:27:40.134832+0000","last_active":"2023-07-12T15:27:49.952389+0000","last_peered":"2023-07-12T15:27:49.952389+0000","last_clean":"2023-07-12T15:27:49.952389+0000","last_became_active":"2023-07-12T15:27:40.134509+0000","last_became_peered":"2023-07-12T15:27:40.134509+0000","last_unstale":"2023-07-12T15:27:49.952389+0000","last_undegraded":"2023-07-12T15:27:49.952389+0000","last_fullsized":"2023-07-12T15:27:49.952389+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:02:32.614487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165547+0000","last_change":"2023-07-12T15:27:42.133184+0000","last_active":"2023-07-12T15:27:50.165547+0000","last_peered":"2023-07-12T15:27:50.165547+0000","last_clean":"2023-07-12T15:27:50.165547+0000","last_became_active":"2023-07-12T15:27:42.132932+0000","last_became_peered":"2023-07-12T15:27:42.132932+0000","last_unstale":"2023-07-12T15:27:50.165547+0000","last_undegraded":"2023-07-12T15:27:50.165547+0000","last_fullsized":"2023-07-12T15:27:50.165547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:25:11.858527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167709+0000","last_change":"2023-07-12T15:27:46.718543+0000","last_active":"2023-07-12T15:27:50.167709+0000","last_peered":"2023-07-12T15:27:50.167709+0000","last_clean":"2023-07-12T15:27:50.167709+0000","last_became_active":"2023-07-12T15:27:46.717706+0000","last_became_peered":"2023-07-12T15:27:46.717706+0000","last_unstale":"2023-07-12T15:27:50.167709+0000","last_undegraded":"2023-07-12T15:27:50.167709+0000","last_fullsized":"2023-07-12T15:27:50.167709+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:57:35.384154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952721+0000","last_change":"2023-07-12T15:27:44.151747+0000","last_active":"2023-07-12T15:27:49.952721+0000","last_peered":"2023-07-12T15:27:49.952721+0000","last_clean":"2023-07-12T15:27:49.952721+0000","last_became_active":"2023-07-12T15:27:44.151415+0000","last_became_peered":"2023-07-12T15:27:44.151415+0000","last_unstale":"2023-07-12T15:27:49.952721+0000","last_undegraded":"2023-07-12T15:27:49.952721+0000","last_fullsized":"2023-07-12T15:27:49.952721+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:53:14.702529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167707+0000","last_change":"2023-07-12T15:27:40.124291+0000","last_active":"2023-07-12T15:27:50.167707+0000","last_peered":"2023-07-12T15:27:50.167707+0000","last_clean":"2023-07-12T15:27:50.167707+0000","last_became_active":"2023-07-12T15:27:40.123913+0000","last_became_peered":"2023-07-12T15:27:40.123913+0000","last_unstale":"2023-07-12T15:27:50.167707+0000","last_undegraded":"2023-07-12T15:27:50.167707+0000","last_fullsized":"2023-07-12T15:27:50.167707+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:05:33.567531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":26,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165602+0000","last_change":"2023-07-12T15:27:42.132617+0000","last_active":"2023-07-12T15:27:50.165602+0000","last_peered":"2023-07-12T15:27:50.165602+0000","last_clean":"2023-07-12T15:27:50.165602+0000","last_became_active":"2023-07-12T15:27:42.132327+0000","last_became_peered":"2023-07-12T15:27:42.132327+0000","last_unstale":"2023-07-12T15:27:50.165602+0000","last_undegraded":"2023-07-12T15:27:50.165602+0000","last_fullsized":"2023-07-12T15:27:50.165602+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:40:47.844771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955866+0000","last_change":"2023-07-12T15:27:46.161301+0000","last_active":"2023-07-12T15:27:49.955866+0000","last_peered":"2023-07-12T15:27:49.955866+0000","last_clean":"2023-07-12T15:27:49.955866+0000","last_became_active":"2023-07-12T15:27:46.160999+0000","last_became_peered":"2023-07-12T15:27:46.160999+0000","last_unstale":"2023-07-12T15:27:49.955866+0000","last_undegraded":"2023-07-12T15:27:49.955866+0000","last_fullsized":"2023-07-12T15:27:49.955866+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:02:24.974403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167053+0000","last_change":"2023-07-12T15:27:44.151957+0000","last_active":"2023-07-12T15:27:50.167053+0000","last_peered":"2023-07-12T15:27:50.167053+0000","last_clean":"2023-07-12T15:27:50.167053+0000","last_became_active":"2023-07-12T15:27:44.151629+0000","last_became_peered":"2023-07-12T15:27:44.151629+0000","last_unstale":"2023-07-12T15:27:50.167053+0000","last_undegraded":"2023-07-12T15:27:50.167053+0000","last_fullsized":"2023-07-12T15:27:50.167053+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:27:28.676471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955867+0000","last_change":"2023-07-12T15:27:40.124203+0000","last_active":"2023-07-12T15:27:49.955867+0000","last_peered":"2023-07-12T15:27:49.955867+0000","last_clean":"2023-07-12T15:27:49.955867+0000","last_became_active":"2023-07-12T15:27:40.123980+0000","last_became_peered":"2023-07-12T15:27:40.123980+0000","last_unstale":"2023-07-12T15:27:49.955867+0000","last_undegraded":"2023-07-12T15:27:49.955867+0000","last_fullsized":"2023-07-12T15:27:49.955867+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:26:07.683170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166504+0000","last_change":"2023-07-12T15:27:42.136284+0000","last_active":"2023-07-12T15:27:50.166504+0000","last_peered":"2023-07-12T15:27:50.166504+0000","last_clean":"2023-07-12T15:27:50.166504+0000","last_became_active":"2023-07-12T15:27:42.135687+0000","last_became_peered":"2023-07-12T15:27:42.135687+0000","last_unstale":"2023-07-12T15:27:50.166504+0000","last_undegraded":"2023-07-12T15:27:50.166504+0000","last_fullsized":"2023-07-12T15:27:50.166504+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:18:59.390617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167282+0000","last_change":"2023-07-12T15:27:46.165713+0000","last_active":"2023-07-12T15:27:50.167282+0000","last_peered":"2023-07-12T15:27:50.167282+0000","last_clean":"2023-07-12T15:27:50.167282+0000","last_became_active":"2023-07-12T15:27:46.165313+0000","last_became_peered":"2023-07-12T15:27:46.165313+0000","last_unstale":"2023-07-12T15:27:50.167282+0000","last_undegraded":"2023-07-12T15:27:50.167282+0000","last_fullsized":"2023-07-12T15:27:50.167282+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:22:47.525502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"60'2","reported_seq":28,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.168012+0000","last_change":"2023-07-12T15:27:40.131073+0000","last_active":"2023-07-12T15:27:50.168012+0000","last_peered":"2023-07-12T15:27:50.168012+0000","last_clean":"2023-07-12T15:27:50.168012+0000","last_became_active":"2023-07-12T15:27:40.130729+0000","last_became_peered":"2023-07-12T15:27:40.130729+0000","last_unstale":"2023-07-12T15:27:50.168012+0000","last_undegraded":"2023-07-12T15:27:50.168012+0000","last_fullsized":"2023-07-12T15:27:50.168012+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:51:07.946411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":40,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167422+0000","last_change":"2023-07-12T15:27:42.140164+0000","last_active":"2023-07-12T15:27:50.167422+0000","last_peered":"2023-07-12T15:27:50.167422+0000","last_clean":"2023-07-12T15:27:50.167422+0000","last_became_active":"2023-07-12T15:27:42.139824+0000","last_became_peered":"2023-07-12T15:27:42.139824+0000","last_unstale":"2023-07-12T15:27:50.167422+0000","last_undegraded":"2023-07-12T15:27:50.167422+0000","last_fullsized":"2023-07-12T15:27:50.167422+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:30:30.583175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955833+0000","last_change":"2023-07-12T15:27:44.144671+0000","last_active":"2023-07-12T15:27:49.955833+0000","last_peered":"2023-07-12T15:27:49.955833+0000","last_clean":"2023-07-12T15:27:49.955833+0000","last_became_active":"2023-07-12T15:27:44.143852+0000","last_became_peered":"2023-07-12T15:27:44.143852+0000","last_unstale":"2023-07-12T15:27:49.955833+0000","last_undegraded":"2023-07-12T15:27:49.955833+0000","last_fullsized":"2023-07-12T15:27:49.955833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:32:53.808907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166542+0000","last_change":"2023-07-12T15:27:46.169482+0000","last_active":"2023-07-12T15:27:50.166542+0000","last_peered":"2023-07-12T15:27:50.166542+0000","last_clean":"2023-07-12T15:27:50.166542+0000","last_became_active":"2023-07-12T15:27:46.168837+0000","last_became_peered":"2023-07-12T15:27:46.168837+0000","last_unstale":"2023-07-12T15:27:50.166542+0000","last_undegraded":"2023-07-12T15:27:50.166542+0000","last_fullsized":"2023-07-12T15:27:50.166542+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:49:10.960844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955323+0000","last_change":"2023-07-12T15:27:40.129384+0000","last_active":"2023-07-12T15:27:49.955323+0000","last_peered":"2023-07-12T15:27:49.955323+0000","last_clean":"2023-07-12T15:27:49.955323+0000","last_became_active":"2023-07-12T15:27:40.129107+0000","last_became_peered":"2023-07-12T15:27:40.129107+0000","last_unstale":"2023-07-12T15:27:49.955323+0000","last_undegraded":"2023-07-12T15:27:49.955323+0000","last_fullsized":"2023-07-12T15:27:49.955323+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:21:16.040857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954842+0000","last_change":"2023-07-12T15:27:42.132498+0000","last_active":"2023-07-12T15:27:49.954842+0000","last_peered":"2023-07-12T15:27:49.954842+0000","last_clean":"2023-07-12T15:27:49.954842+0000","last_became_active":"2023-07-12T15:27:42.132241+0000","last_became_peered":"2023-07-12T15:27:42.132241+0000","last_unstale":"2023-07-12T15:27:49.954842+0000","last_undegraded":"2023-07-12T15:27:49.954842+0000","last_fullsized":"2023-07-12T15:27:49.954842+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:39:34.735107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'11","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:33.814550+0000","last_change":"2023-07-12T15:27:44.155124+0000","last_active":"2023-07-12T15:28:33.814550+0000","last_peered":"2023-07-12T15:28:33.814550+0000","last_clean":"2023-07-12T15:28:33.814550+0000","last_became_active":"2023-07-12T15:27:44.154627+0000","last_became_peered":"2023-07-12T15:27:44.154627+0000","last_unstale":"2023-07-12T15:28:33.814550+0000","last_undegraded":"2023-07-12T15:28:33.814550+0000","last_fullsized":"2023-07-12T15:28:33.814550+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T20:37:59.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166896+0000","last_change":"2023-07-12T15:27:46.164687+0000","last_active":"2023-07-12T15:27:50.166896+0000","last_peered":"2023-07-12T15:27:50.166896+0000","last_clean":"2023-07-12T15:27:50.166896+0000","last_became_active":"2023-07-12T15:27:46.163875+0000","last_became_peered":"2023-07-12T15:27:46.163875+0000","last_unstale":"2023-07-12T15:27:50.166896+0000","last_undegraded":"2023-07-12T15:27:50.166896+0000","last_fullsized":"2023-07-12T15:27:50.166896+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:36:52.918069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952431+0000","last_change":"2023-07-12T15:27:40.130331+0000","last_active":"2023-07-12T15:27:49.952431+0000","last_peered":"2023-07-12T15:27:49.952431+0000","last_clean":"2023-07-12T15:27:49.952431+0000","last_became_active":"2023-07-12T15:27:40.130019+0000","last_became_peered":"2023-07-12T15:27:40.130019+0000","last_unstale":"2023-07-12T15:27:49.952431+0000","last_undegraded":"2023-07-12T15:27:49.952431+0000","last_fullsized":"2023-07-12T15:27:49.952431+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:18:47.396545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'30","reported_seq":80,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:33.814361+0000","last_change":"2023-07-12T15:27:42.133394+0000","last_active":"2023-07-12T15:28:33.814361+0000","last_peered":"2023-07-12T15:28:33.814361+0000","last_clean":"2023-07-12T15:28:33.814361+0000","last_became_active":"2023-07-12T15:27:42.133073+0000","last_became_peered":"2023-07-12T15:27:42.133073+0000","last_unstale":"2023-07-12T15:28:33.814361+0000","last_undegraded":"2023-07-12T15:28:33.814361+0000","last_fullsized":"2023-07-12T15:28:33.814361+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T15:33:49.771223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166475+0000","last_change":"2023-07-12T15:27:44.151055+0000","last_active":"2023-07-12T15:27:50.166475+0000","last_peered":"2023-07-12T15:27:50.166475+0000","last_clean":"2023-07-12T15:27:50.166475+0000","last_became_active":"2023-07-12T15:27:44.150748+0000","last_became_peered":"2023-07-12T15:27:44.150748+0000","last_unstale":"2023-07-12T15:27:50.166475+0000","last_undegraded":"2023-07-12T15:27:50.166475+0000","last_fullsized":"2023-07-12T15:27:50.166475+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:10:09.409726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167891+0000","last_change":"2023-07-12T15:27:46.717308+0000","last_active":"2023-07-12T15:27:50.167891+0000","last_peered":"2023-07-12T15:27:50.167891+0000","last_clean":"2023-07-12T15:27:50.167891+0000","last_became_active":"2023-07-12T15:27:46.717055+0000","last_became_peered":"2023-07-12T15:27:46.717055+0000","last_unstale":"2023-07-12T15:27:50.167891+0000","last_undegraded":"2023-07-12T15:27:50.167891+0000","last_fullsized":"2023-07-12T15:27:50.167891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:43:34.142083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"68'5","reported_seq":80,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:37.935479+0000","last_change":"2023-07-12T15:27:45.145007+0000","last_active":"2023-07-12T15:28:37.935479+0000","last_peered":"2023-07-12T15:28:37.935479+0000","last_clean":"2023-07-12T15:28:37.935479+0000","last_became_active":"2023-07-12T15:27:41.130547+0000","last_became_peered":"2023-07-12T15:27:41.130547+0000","last_unstale":"2023-07-12T15:28:37.935479+0000","last_undegraded":"2023-07-12T15:28:37.935479+0000","last_fullsized":"2023-07-12T15:28:37.935479+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:40.106345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:40.106345+0000","last_clean_scrub_stamp":"2023-07-12T15:27:40.106345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:51:09.700281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034162100000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":54,"num_read_kb":49,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955706+0000","last_change":"2023-07-12T15:27:40.124197+0000","last_active":"2023-07-12T15:27:49.955706+0000","last_peered":"2023-07-12T15:27:49.955706+0000","last_clean":"2023-07-12T15:27:49.955706+0000","last_became_active":"2023-07-12T15:27:40.123876+0000","last_became_peered":"2023-07-12T15:27:40.123876+0000","last_unstale":"2023-07-12T15:27:49.955706+0000","last_undegraded":"2023-07-12T15:27:49.955706+0000","last_fullsized":"2023-07-12T15:27:49.955706+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:41:34.484752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":44,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956805+0000","last_change":"2023-07-12T15:27:42.133473+0000","last_active":"2023-07-12T15:27:49.956805+0000","last_peered":"2023-07-12T15:27:49.956805+0000","last_clean":"2023-07-12T15:27:49.956805+0000","last_became_active":"2023-07-12T15:27:42.133122+0000","last_became_peered":"2023-07-12T15:27:42.133122+0000","last_unstale":"2023-07-12T15:27:49.956805+0000","last_undegraded":"2023-07-12T15:27:49.956805+0000","last_fullsized":"2023-07-12T15:27:49.956805+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:13:29.009977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956114+0000","last_change":"2023-07-12T15:27:44.153052+0000","last_active":"2023-07-12T15:27:49.956114+0000","last_peered":"2023-07-12T15:27:49.956114+0000","last_clean":"2023-07-12T15:27:49.956114+0000","last_became_active":"2023-07-12T15:27:44.152754+0000","last_became_peered":"2023-07-12T15:27:44.152754+0000","last_unstale":"2023-07-12T15:27:49.956114+0000","last_undegraded":"2023-07-12T15:27:49.956114+0000","last_fullsized":"2023-07-12T15:27:49.956114+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:40:57.401230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.953269+0000","last_change":"2023-07-12T15:27:46.165625+0000","last_active":"2023-07-12T15:27:49.953269+0000","last_peered":"2023-07-12T15:27:49.953269+0000","last_clean":"2023-07-12T15:27:49.953269+0000","last_became_active":"2023-07-12T15:27:46.165341+0000","last_became_peered":"2023-07-12T15:27:46.165341+0000","last_unstale":"2023-07-12T15:27:49.953269+0000","last_undegraded":"2023-07-12T15:27:49.953269+0000","last_fullsized":"2023-07-12T15:27:49.953269+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:30:55.751614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"65'2","reported_seq":25,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167125+0000","last_change":"2023-07-12T15:27:45.143925+0000","last_active":"2023-07-12T15:27:50.167125+0000","last_peered":"2023-07-12T15:27:50.167125+0000","last_clean":"2023-07-12T15:27:50.167125+0000","last_became_active":"2023-07-12T15:27:41.130428+0000","last_became_peered":"2023-07-12T15:27:41.130428+0000","last_unstale":"2023-07-12T15:27:50.167125+0000","last_undegraded":"2023-07-12T15:27:50.167125+0000","last_fullsized":"2023-07-12T15:27:50.167125+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:40.106345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:40.106345+0000","last_clean_scrub_stamp":"2023-07-12T15:27:40.106345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:12:37.320950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065463700000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952535+0000","last_change":"2023-07-12T15:27:40.133053+0000","last_active":"2023-07-12T15:27:49.952535+0000","last_peered":"2023-07-12T15:27:49.952535+0000","last_clean":"2023-07-12T15:27:49.952535+0000","last_became_active":"2023-07-12T15:27:40.132735+0000","last_became_peered":"2023-07-12T15:27:40.132735+0000","last_unstale":"2023-07-12T15:27:49.952535+0000","last_undegraded":"2023-07-12T15:27:49.952535+0000","last_fullsized":"2023-07-12T15:27:49.952535+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T16:38:46.103567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"69'128","reported_seq":266,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:13.490058+0000","last_change":"2023-07-12T15:27:30.563173+0000","last_active":"2023-07-12T15:28:13.490058+0000","last_peered":"2023-07-12T15:28:13.490058+0000","last_clean":"2023-07-12T15:28:13.490058+0000","last_became_active":"2023-07-12T15:27:30.562668+0000","last_became_peered":"2023-07-12T15:27:30.562668+0000","last_unstale":"2023-07-12T15:28:13.490058+0000","last_undegraded":"2023-07-12T15:28:13.490058+0000","last_fullsized":"2023-07-12T15:28:13.490058+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:25:40.483044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:25:40.483044+0000","last_clean_scrub_stamp":"2023-07-12T15:25:40.483044+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:26:28.800483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"67'14","reported_seq":39,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166506+0000","last_change":"2023-07-12T15:27:42.132132+0000","last_active":"2023-07-12T15:27:50.166506+0000","last_peered":"2023-07-12T15:27:50.166506+0000","last_clean":"2023-07-12T15:27:50.166506+0000","last_became_active":"2023-07-12T15:27:42.131862+0000","last_became_peered":"2023-07-12T15:27:42.131862+0000","last_unstale":"2023-07-12T15:27:50.166506+0000","last_undegraded":"2023-07-12T15:27:50.166506+0000","last_fullsized":"2023-07-12T15:27:50.166506+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:02:26.304626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166122+0000","last_change":"2023-07-12T15:27:44.157770+0000","last_active":"2023-07-12T15:27:50.166122+0000","last_peered":"2023-07-12T15:27:50.166122+0000","last_clean":"2023-07-12T15:27:50.166122+0000","last_became_active":"2023-07-12T15:27:44.157494+0000","last_became_peered":"2023-07-12T15:27:44.157494+0000","last_unstale":"2023-07-12T15:27:50.166122+0000","last_undegraded":"2023-07-12T15:27:50.166122+0000","last_fullsized":"2023-07-12T15:27:50.166122+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:51:16.040821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954959+0000","last_change":"2023-07-12T15:27:46.170137+0000","last_active":"2023-07-12T15:27:49.954959+0000","last_peered":"2023-07-12T15:27:49.954959+0000","last_clean":"2023-07-12T15:27:49.954959+0000","last_became_active":"2023-07-12T15:27:46.169855+0000","last_became_peered":"2023-07-12T15:27:46.169855+0000","last_unstale":"2023-07-12T15:27:49.954959+0000","last_undegraded":"2023-07-12T15:27:49.954959+0000","last_fullsized":"2023-07-12T15:27:49.954959+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:44:52.967353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166176+0000","last_change":"2023-07-12T15:27:45.146062+0000","last_active":"2023-07-12T15:27:50.166176+0000","last_peered":"2023-07-12T15:27:50.166176+0000","last_clean":"2023-07-12T15:27:50.166176+0000","last_became_active":"2023-07-12T15:27:41.128841+0000","last_became_peered":"2023-07-12T15:27:41.128841+0000","last_unstale":"2023-07-12T15:27:50.166176+0000","last_undegraded":"2023-07-12T15:27:50.166176+0000","last_fullsized":"2023-07-12T15:27:50.166176+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:40.106345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:40.106345+0000","last_clean_scrub_stamp":"2023-07-12T15:27:40.106345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:56:51.194915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075130800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.168153+0000","last_change":"2023-07-12T15:27:40.130214+0000","last_active":"2023-07-12T15:27:50.168153+0000","last_peered":"2023-07-12T15:27:50.168153+0000","last_clean":"2023-07-12T15:27:50.168153+0000","last_became_active":"2023-07-12T15:27:40.129920+0000","last_became_peered":"2023-07-12T15:27:40.129920+0000","last_unstale":"2023-07-12T15:27:50.168153+0000","last_undegraded":"2023-07-12T15:27:50.168153+0000","last_fullsized":"2023-07-12T15:27:50.168153+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:03:47.596959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":35,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166221+0000","last_change":"2023-07-12T15:27:42.138281+0000","last_active":"2023-07-12T15:27:50.166221+0000","last_peered":"2023-07-12T15:27:50.166221+0000","last_clean":"2023-07-12T15:27:50.166221+0000","last_became_active":"2023-07-12T15:27:42.137694+0000","last_became_peered":"2023-07-12T15:27:42.137694+0000","last_unstale":"2023-07-12T15:27:50.166221+0000","last_undegraded":"2023-07-12T15:27:50.166221+0000","last_fullsized":"2023-07-12T15:27:50.166221+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:12:33.813764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.953196+0000","last_change":"2023-07-12T15:27:44.151774+0000","last_active":"2023-07-12T15:27:49.953196+0000","last_peered":"2023-07-12T15:27:49.953196+0000","last_clean":"2023-07-12T15:27:49.953196+0000","last_became_active":"2023-07-12T15:27:44.151468+0000","last_became_peered":"2023-07-12T15:27:44.151468+0000","last_unstale":"2023-07-12T15:27:49.953196+0000","last_undegraded":"2023-07-12T15:27:49.953196+0000","last_fullsized":"2023-07-12T15:27:49.953196+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:21:22.532171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954408+0000","last_change":"2023-07-12T15:27:46.717552+0000","last_active":"2023-07-12T15:27:49.954408+0000","last_peered":"2023-07-12T15:27:49.954408+0000","last_clean":"2023-07-12T15:27:49.954408+0000","last_became_active":"2023-07-12T15:27:46.717184+0000","last_became_peered":"2023-07-12T15:27:46.717184+0000","last_unstale":"2023-07-12T15:27:49.954408+0000","last_undegraded":"2023-07-12T15:27:49.954408+0000","last_fullsized":"2023-07-12T15:27:49.954408+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:53:42.636521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167938+0000","last_change":"2023-07-12T15:27:40.135385+0000","last_active":"2023-07-12T15:27:50.167938+0000","last_peered":"2023-07-12T15:27:50.167938+0000","last_clean":"2023-07-12T15:27:50.167938+0000","last_became_active":"2023-07-12T15:27:40.134949+0000","last_became_peered":"2023-07-12T15:27:40.134949+0000","last_unstale":"2023-07-12T15:27:50.167938+0000","last_undegraded":"2023-07-12T15:27:50.167938+0000","last_fullsized":"2023-07-12T15:27:50.167938+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:05:07.598634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":53,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165547+0000","last_change":"2023-07-12T15:27:42.134809+0000","last_active":"2023-07-12T15:27:50.165547+0000","last_peered":"2023-07-12T15:27:50.165547+0000","last_clean":"2023-07-12T15:27:50.165547+0000","last_became_active":"2023-07-12T15:27:42.134514+0000","last_became_peered":"2023-07-12T15:27:42.134514+0000","last_unstale":"2023-07-12T15:27:50.165547+0000","last_undegraded":"2023-07-12T15:27:50.165547+0000","last_fullsized":"2023-07-12T15:27:50.165547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:54:12.844302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166698+0000","last_change":"2023-07-12T15:27:44.154840+0000","last_active":"2023-07-12T15:27:50.166698+0000","last_peered":"2023-07-12T15:27:50.166698+0000","last_clean":"2023-07-12T15:27:50.166698+0000","last_became_active":"2023-07-12T15:27:44.154537+0000","last_became_peered":"2023-07-12T15:27:44.154537+0000","last_unstale":"2023-07-12T15:27:50.166698+0000","last_undegraded":"2023-07-12T15:27:50.166698+0000","last_fullsized":"2023-07-12T15:27:50.166698+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T03:27:07.097195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166786+0000","last_change":"2023-07-12T15:27:46.163445+0000","last_active":"2023-07-12T15:27:50.166786+0000","last_peered":"2023-07-12T15:27:50.166786+0000","last_clean":"2023-07-12T15:27:50.166786+0000","last_became_active":"2023-07-12T15:27:46.162794+0000","last_became_peered":"2023-07-12T15:27:46.162794+0000","last_unstale":"2023-07-12T15:27:50.166786+0000","last_undegraded":"2023-07-12T15:27:50.166786+0000","last_fullsized":"2023-07-12T15:27:50.166786+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T15:51:01.966202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956429+0000","last_change":"2023-07-12T15:27:40.128114+0000","last_active":"2023-07-12T15:27:49.956429+0000","last_peered":"2023-07-12T15:27:49.956429+0000","last_clean":"2023-07-12T15:27:49.956429+0000","last_became_active":"2023-07-12T15:27:40.127814+0000","last_became_peered":"2023-07-12T15:27:40.127814+0000","last_unstale":"2023-07-12T15:27:49.956429+0000","last_undegraded":"2023-07-12T15:27:49.956429+0000","last_fullsized":"2023-07-12T15:27:49.956429+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:58:48.765074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":49,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165935+0000","last_change":"2023-07-12T15:27:42.134764+0000","last_active":"2023-07-12T15:27:50.165935+0000","last_peered":"2023-07-12T15:27:50.165935+0000","last_clean":"2023-07-12T15:27:50.165935+0000","last_became_active":"2023-07-12T15:27:42.134517+0000","last_became_peered":"2023-07-12T15:27:42.134517+0000","last_unstale":"2023-07-12T15:27:50.165935+0000","last_undegraded":"2023-07-12T15:27:50.165935+0000","last_fullsized":"2023-07-12T15:27:50.165935+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:15:04.623968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167876+0000","last_change":"2023-07-12T15:27:44.152178+0000","last_active":"2023-07-12T15:27:50.167876+0000","last_peered":"2023-07-12T15:27:50.167876+0000","last_clean":"2023-07-12T15:27:50.167876+0000","last_became_active":"2023-07-12T15:27:44.151878+0000","last_became_peered":"2023-07-12T15:27:44.151878+0000","last_unstale":"2023-07-12T15:27:50.167876+0000","last_undegraded":"2023-07-12T15:27:50.167876+0000","last_fullsized":"2023-07-12T15:27:50.167876+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:49:21.080732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167501+0000","last_change":"2023-07-12T15:27:46.720176+0000","last_active":"2023-07-12T15:27:50.167501+0000","last_peered":"2023-07-12T15:27:50.167501+0000","last_clean":"2023-07-12T15:27:50.167501+0000","last_became_active":"2023-07-12T15:27:46.719760+0000","last_became_peered":"2023-07-12T15:27:46.719760+0000","last_unstale":"2023-07-12T15:27:50.167501+0000","last_undegraded":"2023-07-12T15:27:50.167501+0000","last_fullsized":"2023-07-12T15:27:50.167501+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:00:41.076133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167152+0000","last_change":"2023-07-12T15:27:40.129650+0000","last_active":"2023-07-12T15:27:50.167152+0000","last_peered":"2023-07-12T15:27:50.167152+0000","last_clean":"2023-07-12T15:27:50.167152+0000","last_became_active":"2023-07-12T15:27:40.129241+0000","last_became_peered":"2023-07-12T15:27:40.129241+0000","last_unstale":"2023-07-12T15:27:50.167152+0000","last_undegraded":"2023-07-12T15:27:50.167152+0000","last_fullsized":"2023-07-12T15:27:50.167152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:23:17.132392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":38,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167656+0000","last_change":"2023-07-12T15:27:42.138593+0000","last_active":"2023-07-12T15:27:50.167656+0000","last_peered":"2023-07-12T15:27:50.167656+0000","last_clean":"2023-07-12T15:27:50.167656+0000","last_became_active":"2023-07-12T15:27:42.138275+0000","last_became_peered":"2023-07-12T15:27:42.138275+0000","last_unstale":"2023-07-12T15:27:50.167656+0000","last_undegraded":"2023-07-12T15:27:50.167656+0000","last_fullsized":"2023-07-12T15:27:50.167656+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:28:49.929423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'11","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:33.814379+0000","last_change":"2023-07-12T15:27:44.155072+0000","last_active":"2023-07-12T15:28:33.814379+0000","last_peered":"2023-07-12T15:28:33.814379+0000","last_clean":"2023-07-12T15:28:33.814379+0000","last_became_active":"2023-07-12T15:27:44.154800+0000","last_became_peered":"2023-07-12T15:27:44.154800+0000","last_unstale":"2023-07-12T15:28:33.814379+0000","last_undegraded":"2023-07-12T15:28:33.814379+0000","last_fullsized":"2023-07-12T15:28:33.814379+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:25:04.242576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166978+0000","last_change":"2023-07-12T15:27:46.718096+0000","last_active":"2023-07-12T15:27:50.166978+0000","last_peered":"2023-07-12T15:27:50.166978+0000","last_clean":"2023-07-12T15:27:50.166978+0000","last_became_active":"2023-07-12T15:27:46.717778+0000","last_became_peered":"2023-07-12T15:27:46.717778+0000","last_unstale":"2023-07-12T15:27:50.166978+0000","last_undegraded":"2023-07-12T15:27:50.166978+0000","last_fullsized":"2023-07-12T15:27:50.166978+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:54:33.593328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.953066+0000","last_change":"2023-07-12T15:27:44.153649+0000","last_active":"2023-07-12T15:27:49.953066+0000","last_peered":"2023-07-12T15:27:49.953066+0000","last_clean":"2023-07-12T15:27:49.953066+0000","last_became_active":"2023-07-12T15:27:44.153297+0000","last_became_peered":"2023-07-12T15:27:44.153297+0000","last_unstale":"2023-07-12T15:27:49.953066+0000","last_undegraded":"2023-07-12T15:27:49.953066+0000","last_fullsized":"2023-07-12T15:27:49.953066+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:40:58.363613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.168272+0000","last_change":"2023-07-12T15:27:40.130023+0000","last_active":"2023-07-12T15:27:50.168272+0000","last_peered":"2023-07-12T15:27:50.168272+0000","last_clean":"2023-07-12T15:27:50.168272+0000","last_became_active":"2023-07-12T15:27:40.129662+0000","last_became_peered":"2023-07-12T15:27:40.129662+0000","last_unstale":"2023-07-12T15:27:50.168272+0000","last_undegraded":"2023-07-12T15:27:50.168272+0000","last_fullsized":"2023-07-12T15:27:50.168272+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:46:44.335318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166796+0000","last_change":"2023-07-12T15:27:42.140160+0000","last_active":"2023-07-12T15:27:50.166796+0000","last_peered":"2023-07-12T15:27:50.166796+0000","last_clean":"2023-07-12T15:27:50.166796+0000","last_became_active":"2023-07-12T15:27:42.139927+0000","last_became_peered":"2023-07-12T15:27:42.139927+0000","last_unstale":"2023-07-12T15:27:50.166796+0000","last_undegraded":"2023-07-12T15:27:50.166796+0000","last_fullsized":"2023-07-12T15:27:50.166796+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:42:04.716802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952668+0000","last_change":"2023-07-12T15:27:46.166865+0000","last_active":"2023-07-12T15:27:49.952668+0000","last_peered":"2023-07-12T15:27:49.952668+0000","last_clean":"2023-07-12T15:27:49.952668+0000","last_became_active":"2023-07-12T15:27:46.166552+0000","last_became_peered":"2023-07-12T15:27:46.166552+0000","last_unstale":"2023-07-12T15:27:49.952668+0000","last_undegraded":"2023-07-12T15:27:49.952668+0000","last_fullsized":"2023-07-12T15:27:49.952668+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:48:27.558516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'11","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:33.814534+0000","last_change":"2023-07-12T15:27:44.148480+0000","last_active":"2023-07-12T15:28:33.814534+0000","last_peered":"2023-07-12T15:28:33.814534+0000","last_clean":"2023-07-12T15:28:33.814534+0000","last_became_active":"2023-07-12T15:27:44.148136+0000","last_became_peered":"2023-07-12T15:27:44.148136+0000","last_unstale":"2023-07-12T15:28:33.814534+0000","last_undegraded":"2023-07-12T15:28:33.814534+0000","last_fullsized":"2023-07-12T15:28:33.814534+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:28:53.776463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954638+0000","last_change":"2023-07-12T15:27:40.129260+0000","last_active":"2023-07-12T15:27:49.954638+0000","last_peered":"2023-07-12T15:27:49.954638+0000","last_clean":"2023-07-12T15:27:49.954638+0000","last_became_active":"2023-07-12T15:27:40.129004+0000","last_became_peered":"2023-07-12T15:27:40.129004+0000","last_unstale":"2023-07-12T15:27:49.954638+0000","last_undegraded":"2023-07-12T15:27:49.954638+0000","last_fullsized":"2023-07-12T15:27:49.954638+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:38:35.484831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956803+0000","last_change":"2023-07-12T15:27:42.135255+0000","last_active":"2023-07-12T15:27:49.956803+0000","last_peered":"2023-07-12T15:27:49.956803+0000","last_clean":"2023-07-12T15:27:49.956803+0000","last_became_active":"2023-07-12T15:27:42.134833+0000","last_became_peered":"2023-07-12T15:27:42.134833+0000","last_unstale":"2023-07-12T15:27:49.956803+0000","last_undegraded":"2023-07-12T15:27:49.956803+0000","last_fullsized":"2023-07-12T15:27:49.956803+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:47:00.249711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166890+0000","last_change":"2023-07-12T15:27:46.156648+0000","last_active":"2023-07-12T15:27:50.166890+0000","last_peered":"2023-07-12T15:27:50.166890+0000","last_clean":"2023-07-12T15:27:50.166890+0000","last_became_active":"2023-07-12T15:27:46.156356+0000","last_became_peered":"2023-07-12T15:27:46.156356+0000","last_unstale":"2023-07-12T15:27:50.166890+0000","last_undegraded":"2023-07-12T15:27:50.166890+0000","last_fullsized":"2023-07-12T15:27:50.166890+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:54:06.704264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'11","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:33.814380+0000","last_change":"2023-07-12T15:27:44.153247+0000","last_active":"2023-07-12T15:28:33.814380+0000","last_peered":"2023-07-12T15:28:33.814380+0000","last_clean":"2023-07-12T15:28:33.814380+0000","last_became_active":"2023-07-12T15:27:44.152959+0000","last_became_peered":"2023-07-12T15:27:44.152959+0000","last_unstale":"2023-07-12T15:28:33.814380+0000","last_undegraded":"2023-07-12T15:28:33.814380+0000","last_fullsized":"2023-07-12T15:28:33.814380+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:33:06.982607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.954602+0000","last_change":"2023-07-12T15:27:40.131849+0000","last_active":"2023-07-12T15:27:49.954602+0000","last_peered":"2023-07-12T15:27:49.954602+0000","last_clean":"2023-07-12T15:27:49.954602+0000","last_became_active":"2023-07-12T15:27:40.131588+0000","last_became_peered":"2023-07-12T15:27:40.131588+0000","last_unstale":"2023-07-12T15:27:49.954602+0000","last_undegraded":"2023-07-12T15:27:49.954602+0000","last_fullsized":"2023-07-12T15:27:49.954602+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:00:12.318976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":32,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166937+0000","last_change":"2023-07-12T15:27:42.138771+0000","last_active":"2023-07-12T15:27:50.166937+0000","last_peered":"2023-07-12T15:27:50.166937+0000","last_clean":"2023-07-12T15:27:50.166937+0000","last_became_active":"2023-07-12T15:27:42.138522+0000","last_became_peered":"2023-07-12T15:27:42.138522+0000","last_unstale":"2023-07-12T15:27:50.166937+0000","last_undegraded":"2023-07-12T15:27:50.166937+0000","last_fullsized":"2023-07-12T15:27:50.166937+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:22:02.312269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956211+0000","last_change":"2023-07-12T15:27:46.166137+0000","last_active":"2023-07-12T15:27:49.956211+0000","last_peered":"2023-07-12T15:27:49.956211+0000","last_clean":"2023-07-12T15:27:49.956211+0000","last_became_active":"2023-07-12T15:27:46.165854+0000","last_became_peered":"2023-07-12T15:27:46.165854+0000","last_unstale":"2023-07-12T15:27:49.956211+0000","last_undegraded":"2023-07-12T15:27:49.956211+0000","last_fullsized":"2023-07-12T15:27:49.956211+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:26:21.892927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955216+0000","last_change":"2023-07-12T15:27:44.146467+0000","last_active":"2023-07-12T15:27:49.955216+0000","last_peered":"2023-07-12T15:27:49.955216+0000","last_clean":"2023-07-12T15:27:49.955216+0000","last_became_active":"2023-07-12T15:27:44.146136+0000","last_became_peered":"2023-07-12T15:27:44.146136+0000","last_unstale":"2023-07-12T15:27:49.955216+0000","last_undegraded":"2023-07-12T15:27:49.955216+0000","last_fullsized":"2023-07-12T15:27:49.955216+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:39:29.266566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167313+0000","last_change":"2023-07-12T15:27:40.134992+0000","last_active":"2023-07-12T15:27:50.167313+0000","last_peered":"2023-07-12T15:27:50.167313+0000","last_clean":"2023-07-12T15:27:50.167313+0000","last_became_active":"2023-07-12T15:27:40.134695+0000","last_became_peered":"2023-07-12T15:27:40.134695+0000","last_unstale":"2023-07-12T15:27:50.167313+0000","last_undegraded":"2023-07-12T15:27:50.167313+0000","last_fullsized":"2023-07-12T15:27:50.167313+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:26:00.079738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":45,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.167058+0000","last_change":"2023-07-12T15:27:42.140726+0000","last_active":"2023-07-12T15:27:50.167058+0000","last_peered":"2023-07-12T15:27:50.167058+0000","last_clean":"2023-07-12T15:27:50.167058+0000","last_became_active":"2023-07-12T15:27:42.140452+0000","last_became_peered":"2023-07-12T15:27:42.140452+0000","last_unstale":"2023-07-12T15:27:50.167058+0000","last_undegraded":"2023-07-12T15:27:50.167058+0000","last_fullsized":"2023-07-12T15:27:50.167058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T21:55:35.249407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.168086+0000","last_change":"2023-07-12T15:27:46.163623+0000","last_active":"2023-07-12T15:27:50.168086+0000","last_peered":"2023-07-12T15:27:50.168086+0000","last_clean":"2023-07-12T15:27:50.168086+0000","last_became_active":"2023-07-12T15:27:46.163345+0000","last_became_peered":"2023-07-12T15:27:46.163345+0000","last_unstale":"2023-07-12T15:27:50.168086+0000","last_undegraded":"2023-07-12T15:27:50.168086+0000","last_fullsized":"2023-07-12T15:27:50.168086+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T02:16:38.639243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955948+0000","last_change":"2023-07-12T15:27:44.153904+0000","last_active":"2023-07-12T15:27:49.955948+0000","last_peered":"2023-07-12T15:27:49.955948+0000","last_clean":"2023-07-12T15:27:49.955948+0000","last_became_active":"2023-07-12T15:27:44.153612+0000","last_became_peered":"2023-07-12T15:27:44.153612+0000","last_unstale":"2023-07-12T15:27:49.955948+0000","last_undegraded":"2023-07-12T15:27:49.955948+0000","last_fullsized":"2023-07-12T15:27:49.955948+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T17:39:37.439894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956103+0000","last_change":"2023-07-12T15:27:40.130234+0000","last_active":"2023-07-12T15:27:49.956103+0000","last_peered":"2023-07-12T15:27:49.956103+0000","last_clean":"2023-07-12T15:27:49.956103+0000","last_became_active":"2023-07-12T15:27:40.129916+0000","last_became_peered":"2023-07-12T15:27:40.129916+0000","last_unstale":"2023-07-12T15:27:49.956103+0000","last_undegraded":"2023-07-12T15:27:49.956103+0000","last_fullsized":"2023-07-12T15:27:49.956103+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:26:58.825905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":48,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.166436+0000","last_change":"2023-07-12T15:27:42.132070+0000","last_active":"2023-07-12T15:27:50.166436+0000","last_peered":"2023-07-12T15:27:50.166436+0000","last_clean":"2023-07-12T15:27:50.166436+0000","last_became_active":"2023-07-12T15:27:42.131822+0000","last_became_peered":"2023-07-12T15:27:42.131822+0000","last_unstale":"2023-07-12T15:27:50.166436+0000","last_undegraded":"2023-07-12T15:27:50.166436+0000","last_fullsized":"2023-07-12T15:27:50.166436+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T22:38:21.874493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165622+0000","last_change":"2023-07-12T15:27:46.169164+0000","last_active":"2023-07-12T15:27:50.165622+0000","last_peered":"2023-07-12T15:27:50.165622+0000","last_clean":"2023-07-12T15:27:50.165622+0000","last_became_active":"2023-07-12T15:27:46.168858+0000","last_became_peered":"2023-07-12T15:27:46.168858+0000","last_unstale":"2023-07-12T15:27:50.165622+0000","last_undegraded":"2023-07-12T15:27:50.165622+0000","last_fullsized":"2023-07-12T15:27:50.165622+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T00:53:33.591948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.955833+0000","last_change":"2023-07-12T15:27:44.149152+0000","last_active":"2023-07-12T15:27:49.955833+0000","last_peered":"2023-07-12T15:27:49.955833+0000","last_clean":"2023-07-12T15:27:49.955833+0000","last_became_active":"2023-07-12T15:27:44.148823+0000","last_became_peered":"2023-07-12T15:27:44.148823+0000","last_unstale":"2023-07-12T15:27:49.955833+0000","last_undegraded":"2023-07-12T15:27:49.955833+0000","last_fullsized":"2023-07-12T15:27:49.955833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-14T01:29:02.458613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"60'1","reported_seq":24,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956185+0000","last_change":"2023-07-12T15:27:40.123921+0000","last_active":"2023-07-12T15:27:49.956185+0000","last_peered":"2023-07-12T15:27:49.956185+0000","last_clean":"2023-07-12T15:27:49.956185+0000","last_became_active":"2023-07-12T15:27:40.123639+0000","last_became_peered":"2023-07-12T15:27:40.123639+0000","last_unstale":"2023-07-12T15:27:49.956185+0000","last_undegraded":"2023-07-12T15:27:49.956185+0000","last_fullsized":"2023-07-12T15:27:49.956185+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T19:59:53.557008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":33,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:50.165408+0000","last_change":"2023-07-12T15:27:42.131559+0000","last_active":"2023-07-12T15:27:50.165408+0000","last_peered":"2023-07-12T15:27:50.165408+0000","last_clean":"2023-07-12T15:27:50.165408+0000","last_became_active":"2023-07-12T15:27:42.131301+0000","last_became_peered":"2023-07-12T15:27:42.131301+0000","last_unstale":"2023-07-12T15:27:50.165408+0000","last_undegraded":"2023-07-12T15:27:50.165408+0000","last_fullsized":"2023-07-12T15:27:50.165408+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:34:47.872222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952345+0000","last_change":"2023-07-12T15:27:46.718018+0000","last_active":"2023-07-12T15:27:49.952345+0000","last_peered":"2023-07-12T15:27:49.952345+0000","last_clean":"2023-07-12T15:27:49.952345+0000","last_became_active":"2023-07-12T15:27:46.717731+0000","last_became_peered":"2023-07-12T15:27:46.717731+0000","last_unstale":"2023-07-12T15:27:49.952345+0000","last_undegraded":"2023-07-12T15:27:49.952345+0000","last_fullsized":"2023-07-12T15:27:49.952345+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:45.134610+0000","last_clean_scrub_stamp":"2023-07-12T15:27:45.134610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:24:33.208742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.956304+0000","last_change":"2023-07-12T15:27:40.132269+0000","last_active":"2023-07-12T15:27:49.956304+0000","last_peered":"2023-07-12T15:27:49.956304+0000","last_clean":"2023-07-12T15:27:49.956304+0000","last_became_active":"2023-07-12T15:27:40.131890+0000","last_became_peered":"2023-07-12T15:27:40.131890+0000","last_unstale":"2023-07-12T15:27:49.956304+0000","last_undegraded":"2023-07-12T15:27:49.956304+0000","last_fullsized":"2023-07-12T15:27:49.956304+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:39.104561+0000","last_clean_scrub_stamp":"2023-07-12T15:27:39.104561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T23:50:52.926439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-07-12T15:27:49.952270+0000","last_change":"2023-07-12T15:27:42.137519+0000","last_active":"2023-07-12T15:27:49.952270+0000","last_peered":"2023-07-12T15:27:49.952270+0000","last_clean":"2023-07-12T15:27:49.952270+0000","last_became_active":"2023-07-12T15:27:42.137252+0000","last_became_peered":"2023-07-12T15:27:42.137252+0000","last_unstale":"2023-07-12T15:27:49.952270+0000","last_undegraded":"2023-07-12T15:27:49.952270+0000","last_fullsized":"2023-07-12T15:27:49.952270+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:41.112973+0000","last_clean_scrub_stamp":"2023-07-12T15:27:41.112973+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:52:36.902814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'11","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-07-12T15:28:33.814530+0000","last_change":"2023-07-12T15:27:44.156440+0000","last_active":"2023-07-12T15:28:33.814530+0000","last_peered":"2023-07-12T15:28:33.814530+0000","last_clean":"2023-07-12T15:28:33.814530+0000","last_became_active":"2023-07-12T15:27:44.156149+0000","last_became_peered":"2023-07-12T15:27:44.156149+0000","last_unstale":"2023-07-12T15:28:33.814530+0000","last_undegraded":"2023-07-12T15:28:33.814530+0000","last_fullsized":"2023-07-12T15:28:33.814530+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-12T15:27:43.131037+0000","last_clean_scrub_stamp":"2023-07-12T15:27:43.131037+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-13T18:25:54.068870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":49,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1364,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234000,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28192,"kb_used_data":1240,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716960,"statfs":{"total":95995035648,"available":95966167040,"internally_reserved":0,"allocated":1269760,"data_stored":853900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.57599999999999996}]},{"osd":1,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.626}]},{"osd":2,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.60599999999999998}]},{"osd":3,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.407,"5min":3.407,"15min":3.407},"last":0.52800000000000002}]},{"osd":4,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.505}]},{"osd":5,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64300000000000002}]},{"osd":6,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.65500000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495636,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1249280,"data_stored":853333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.435},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.34300000000000003}]},{"osd":1,"last update":"Wed Jul 12 15:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.32000000000000001}]},{"osd":2,"last update":"Wed Jul 12 15:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.628}]},{"osd":3,"last update":"Wed Jul 12 15:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.495}]},{"osd":4,"last update":"Wed Jul 12 15:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.56399999999999995}]},{"osd":5,"last update":"Wed Jul 12 15:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.51400000000000001}]},{"osd":7,"last update":"Wed Jul 12 15:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.61599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27656,"kb_used_data":704,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":720896,"data_stored":265215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34999999999999998,"15min":0.34200000000000003},"min":{"1min":0.23699999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.48399999999999999,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.373},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.38300000000000001,"15min":0.371},"min":{"1min":0.24299999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.41499999999999998}]},{"osd":2,"last update":"Wed Jul 12 15:27:46 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.39100000000000001,"15min":0.39300000000000002},"min":{"1min":0.28499999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.50900000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53500000000000003,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.48099999999999998}]},{"osd":3,"last update":"Wed Jul 12 15:28:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.443},{"interface":"front","average":{"1min":0.436,"5min":0.44,"15min":0.441},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.43099999999999999}]},{"osd":4,"last update":"Wed Jul 12 15:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.44500000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.72599999999999998}]},{"osd":5,"last update":"Wed Jul 12 15:27:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.626},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55500000000000005}]},{"osd":6,"last update":"Wed Jul 12 15:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.59199999999999997}]},{"osd":7,"last update":"Wed Jul 12 15:28:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.69899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738415,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1273856,"data_stored":854471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jul 12 15:28:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.33400000000000002,"15min":0.32000000000000001},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.33700000000000002},"min":{"1min":0.215,"5min":0.214,"15min":0.214},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.35599999999999998}]},{"osd":2,"last update":"Wed Jul 12 15:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.38700000000000001,"15min":0.38800000000000001},"min":{"1min":0.27900000000000003,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.49399999999999999,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.373},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39400000000000002,"15min":0.39100000000000001},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.42099999999999999}]},{"osd":3,"last update":"Wed Jul 12 15:28:05 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.438,"15min":0.436},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42899999999999999,"15min":0.43099999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.38}]},{"osd":4,"last update":"Wed Jul 12 15:28:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.78300000000000003}]},{"osd":5,"last update":"Wed Jul 12 15:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.73999999999999999}]},{"osd":6,"last update":"Wed Jul 12 15:28:05 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.752},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.59899999999999998}]},{"osd":7,"last update":"Wed Jul 12 15:28:31 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.72099999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411366,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27596,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717556,"statfs":{"total":95995035648,"available":95966777344,"internally_reserved":0,"allocated":659456,"data_stored":263633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:27:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.377},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.377,"15min":0.375},"min":{"1min":0.311,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.25800000000000001}]},{"osd":1,"last update":"Wed Jul 12 15:27:42 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.39000000000000001,"15min":0.38500000000000001},"min":{"1min":0.253,"5min":0.245,"15min":0.245},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.41499999999999998,"15min":0.41299999999999998},"min":{"1min":0.29599999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.34200000000000003}]},{"osd":3,"last update":"Wed Jul 12 15:28:08 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.47399999999999998,"15min":0.48099999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71799999999999997,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.307},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45900000000000002,"15min":0.46700000000000003},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.29199999999999998}]},{"osd":4,"last update":"Wed Jul 12 15:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.65000000000000002,"15min":0.64000000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1110000000000002,"5min":3.1110000000000002,"15min":3.1110000000000002},"last":0.495},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.66700000000000004,"15min":0.66200000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.6960000000000002,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":0.50900000000000001}]},{"osd":5,"last update":"Wed Jul 12 15:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.53100000000000003}]},{"osd":6,"last update":"Wed Jul 12 15:28:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.51900000000000002}]},{"osd":7,"last update":"Wed Jul 12 15:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":3.0880000000000001,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":2.6349999999999998,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":0.435}]}]},{"osd":3,"up_from":25,"seq":107374182434,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27636,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717516,"statfs":{"total":95995035648,"available":95966736384,"internally_reserved":0,"allocated":700416,"data_stored":264993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.39600000000000002,"15min":0.40200000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.55100000000000005,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.222},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.443}]},{"osd":1,"last update":"Wed Jul 12 15:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53000000000000003,"5min":0.63,"15min":0.63},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.54200000000000004,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.20899999999999999}]},{"osd":2,"last update":"Wed Jul 12 15:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.46400000000000002,"15min":0.46800000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.56699999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.435,"15min":0.44},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.312}]},{"osd":4,"last update":"Wed Jul 12 15:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.41599999999999998}]},{"osd":5,"last update":"Wed Jul 12 15:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.42499999999999999}]},{"osd":6,"last update":"Wed Jul 12 15:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.435}]},{"osd":7,"last update":"Wed Jul 12 15:28:29 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.53600000000000003}]}]},{"osd":4,"up_from":30,"seq":128849018910,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27620,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717532,"statfs":{"total":95995035648,"available":95966752768,"internally_reserved":0,"allocated":684032,"data_stored":264041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.57299999999999995,"15min":0.58699999999999997},"min":{"1min":0.35499999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.79300000000000004,"5min":1.268,"15min":1.268},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.39500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80200000000000005,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.34599999999999997}]},{"osd":1,"last update":"Wed Jul 12 15:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.376,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68100000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.61299999999999999,"15min":0.626},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.80100000000000005,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.40000000000000002}]},{"osd":2,"last update":"Wed Jul 12 15:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.58399999999999996,"15min":0.59499999999999997},"min":{"1min":0.434,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63800000000000001,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.57199999999999995}]},{"osd":3,"last update":"Wed Jul 12 15:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.58299999999999996,"15min":0.59499999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85399999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.499},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58999999999999997,"15min":0.59799999999999998},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76700000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.44}]},{"osd":5,"last update":"Wed Jul 12 15:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.61899999999999999}]},{"osd":6,"last update":"Wed Jul 12 15:28:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.64400000000000002}]},{"osd":7,"last update":"Wed Jul 12 15:28:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.63400000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757272,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27600,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":663552,"data_stored":264056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 12 15:27:43 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.7149999999999999,"5min":4.7149999999999999,"15min":4.7149999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.53400000000000003}]},{"osd":1,"last update":"Wed Jul 12 15:27:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":4.4710000000000001,"5min":4.4710000000000001,"15min":4.4710000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.7170000000000001,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":0.45900000000000002}]},{"osd":2,"last update":"Wed Jul 12 15:27:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.5659999999999998,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":6.1349999999999998,"5min":6.1349999999999998,"15min":6.1349999999999998},"last":0.55700000000000005}]},{"osd":3,"last update":"Wed Jul 12 15:27:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":3.5859999999999999,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.47999999999999998}]},{"osd":4,"last update":"Wed Jul 12 15:27:43 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.5880000000000001,"5min":2.5880000000000001,"15min":2.5880000000000001},"last":0.499},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.4180000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":0.63}]},{"osd":6,"last update":"Wed Jul 12 15:28:08 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.9459999999999997,"5min":4.9459999999999997,"15min":4.9459999999999997},"last":0.68500000000000005}]},{"osd":7,"last update":"Wed Jul 12 15:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":3.3610000000000002,"5min":3.3610000000000002,"15min":3.3610000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.633,"5min":4.633,"15min":4.633},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-12T15:28:41.020 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-07-12T15:28:41.020 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-07-12T15:28:41.021 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-07-12T15:28:41.021 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b1bd054042b7797ba5b70d145a1da31693dda494 shell --fsid dc75a490-20c7-11ee-9b31-001a4aab830c -- ceph health --format=json 2023-07-12T15:28:41.570 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/mon.a/config 2023-07-12T15:28:42.634 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:42 smithi069 ceph-mon[102777]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-12T15:28:42.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:42 smithi069 ceph-mon[102777]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:42.635 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:42 smithi069 ceph-mon[94241]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-12T15:28:42.635 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:42 smithi069 ceph-mon[94241]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:42.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:42 smithi145 ceph-mon[95801]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-12T15:28:42.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:42 smithi145 ceph-mon[95801]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:42.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.849+0000 7fac89333700 1 -- 172.21.15.69:0/2357672889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac84101610 msgr2=0x7fac8410c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:42.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.849+0000 7fac89333700 1 --2- 172.21.15.69:0/2357672889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac84101610 0x7fac8410c000 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fac7c00b0d0 tx=0x7fac7c00b3e0 comp rx=0 tx=0).stop 2023-07-12T15:28:42.851 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.849+0000 7fac89333700 1 -- 172.21.15.69:0/2357672889 shutdown_connections 2023-07-12T15:28:42.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.849+0000 7fac89333700 1 --2- 172.21.15.69:0/2357672889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac84101610 0x7fac8410c000 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:42.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.849+0000 7fac89333700 1 --2- 172.21.15.69:0/2357672889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac840ff400 0x7fac840ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:42.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.849+0000 7fac89333700 1 --2- 172.21.15.69:0/2357672889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac84100c60 0x7fac841010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:42.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.849+0000 7fac89333700 1 -- 172.21.15.69:0/2357672889 >> 172.21.15.69:0/2357672889 conn(0x7fac84076150 msgr2=0x7fac84076550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:42.852 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.849+0000 7fac89333700 1 -- 172.21.15.69:0/2357672889 shutdown_connections 2023-07-12T15:28:42.853 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.849+0000 7fac89333700 1 -- 172.21.15.69:0/2357672889 wait complete. 2023-07-12T15:28:42.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.849+0000 7fac89333700 1 Processor -- start 2023-07-12T15:28:42.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac89333700 1 -- start start 2023-07-12T15:28:42.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac89333700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac840ff400 0x7fac84101e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:42.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac89333700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac84100c60 0x7fac84102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:42.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac89333700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac84101610 0x7fac84102900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:42.855 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac89333700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac841a3f70 con 0x7fac84101610 2023-07-12T15:28:42.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac89333700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac841a40e0 con 0x7fac840ff400 2023-07-12T15:28:42.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac89333700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac841a42a0 con 0x7fac84100c60 2023-07-12T15:28:42.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac837fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac84101610 0x7fac84102900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:42.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac837fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac84101610 0x7fac84102900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41546/0 (socket says 172.21.15.69:41546) 2023-07-12T15:28:42.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac837fe700 1 -- 172.21.15.69:0/1361724873 learned_addr learned my addr 172.21.15.69:0/1361724873 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:28:42.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac82ffd700 1 --2- 172.21.15.69:0/1361724873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac840ff400 0x7fac84101e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:42.856 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac827fc700 1 --2- 172.21.15.69:0/1361724873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac84100c60 0x7fac84102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:42.857 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac837fe700 1 -- 172.21.15.69:0/1361724873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac84100c60 msgr2=0x7fac84102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:42.857 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac837fe700 1 --2- 172.21.15.69:0/1361724873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac84100c60 0x7fac84102380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:42.857 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac837fe700 1 -- 172.21.15.69:0/1361724873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac840ff400 msgr2=0x7fac84101e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:42.857 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac837fe700 1 --2- 172.21.15.69:0/1361724873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac840ff400 0x7fac84101e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:42.857 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac837fe700 1 -- 172.21.15.69:0/1361724873 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c009520 con 0x7fac84101610 2023-07-12T15:28:42.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.850+0000 7fac837fe700 1 --2- 172.21.15.69:0/1361724873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac84101610 0x7fac84102900 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fac7c008ed0 tx=0x7fac7c008f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:42.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.851+0000 7fac7bfff700 1 -- 172.21.15.69:0/1361724873 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c00e040 con 0x7fac84101610 2023-07-12T15:28:42.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.851+0000 7fac7bfff700 1 -- 172.21.15.69:0/1361724873 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fac7c003e90 con 0x7fac84101610 2023-07-12T15:28:42.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.851+0000 7fac7bfff700 1 -- 172.21.15.69:0/1361724873 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c01b6c0 con 0x7fac84101610 2023-07-12T15:28:42.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.851+0000 7fac89333700 1 -- 172.21.15.69:0/1361724873 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac7c009d00 con 0x7fac84101610 2023-07-12T15:28:42.858 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.851+0000 7fac89333700 1 -- 172.21.15.69:0/1361724873 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac841a47d0 con 0x7fac84101610 2023-07-12T15:28:42.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.852+0000 7fac89333700 1 -- 172.21.15.69:0/1361724873 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac8404e800 con 0x7fac84101610 2023-07-12T15:28:42.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.852+0000 7fac7bfff700 1 -- 172.21.15.69:0/1361724873 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7fac7c003a00 con 0x7fac84101610 2023-07-12T15:28:42.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.852+0000 7fac7bfff700 1 --2- 172.21.15.69:0/1361724873 >> v2:172.21.15.69:6800/1242716914 conn(0x7fac7006c810 0x7fac7006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:28:42.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.853+0000 7fac7bfff700 1 -- 172.21.15.69:0/1361724873 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fac7c08ccb0 con 0x7fac84101610 2023-07-12T15:28:42.859 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.853+0000 7fac82ffd700 1 --2- 172.21.15.69:0/1361724873 >> v2:172.21.15.69:6800/1242716914 conn(0x7fac7006c810 0x7fac7006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:28:42.860 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.853+0000 7fac82ffd700 1 --2- 172.21.15.69:0/1361724873 >> v2:172.21.15.69:6800/1242716914 conn(0x7fac7006c810 0x7fac7006ecc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fac6c009df0 tx=0x7fac6c009260 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:28:42.860 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:42.857+0000 7fac7bfff700 1 -- 172.21.15.69:0/1361724873 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fac7c05aac0 con 0x7fac84101610 2023-07-12T15:28:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.085+0000 7fac89333700 1 -- 172.21.15.69:0/1361724873 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fac841a49f0 con 0x7fac84101610 2023-07-12T15:28:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.086+0000 7fac7bfff700 1 -- 172.21.15.69:0/1361724873 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fac7c05a650 con 0x7fac84101610 2023-07-12T15:28:43.088 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-07-12T15:28:43.089 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-12T15:28:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.088+0000 7fac89333700 1 -- 172.21.15.69:0/1361724873 >> v2:172.21.15.69:6800/1242716914 conn(0x7fac7006c810 msgr2=0x7fac7006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.088+0000 7fac89333700 1 --2- 172.21.15.69:0/1361724873 >> v2:172.21.15.69:6800/1242716914 conn(0x7fac7006c810 0x7fac7006ecc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fac6c009df0 tx=0x7fac6c009260 comp rx=0 tx=0).stop 2023-07-12T15:28:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.088+0000 7fac89333700 1 -- 172.21.15.69:0/1361724873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac84101610 msgr2=0x7fac84102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:28:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.088+0000 7fac89333700 1 --2- 172.21.15.69:0/1361724873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac84101610 0x7fac84102900 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fac7c008ed0 tx=0x7fac7c008f00 comp rx=0 tx=0).stop 2023-07-12T15:28:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.088+0000 7fac89333700 1 -- 172.21.15.69:0/1361724873 shutdown_connections 2023-07-12T15:28:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.088+0000 7fac89333700 1 --2- 172.21.15.69:0/1361724873 >> v2:172.21.15.69:6800/1242716914 conn(0x7fac7006c810 0x7fac7006ecc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.088+0000 7fac89333700 1 --2- 172.21.15.69:0/1361724873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac84101610 0x7fac84102900 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.088+0000 7fac89333700 1 --2- 172.21.15.69:0/1361724873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac840ff400 0x7fac84101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.088+0000 7fac89333700 1 --2- 172.21.15.69:0/1361724873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac84100c60 0x7fac84102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:28:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.088+0000 7fac89333700 1 -- 172.21.15.69:0/1361724873 >> 172.21.15.69:0/1361724873 conn(0x7fac84076150 msgr2=0x7fac84103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:28:43.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.088+0000 7fac89333700 1 -- 172.21.15.69:0/1361724873 shutdown_connections 2023-07-12T15:28:43.092 INFO:teuthology.orchestra.run.smithi069.stderr:2023-07-12T15:28:43.088+0000 7fac89333700 1 -- 172.21.15.69:0/1361724873 wait complete. 2023-07-12T15:28:43.416 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-07-12T15:28:43.416 INFO:tasks.cephadm:Setup complete, yielding 2023-07-12T15:28:43.417 INFO:teuthology.run_tasks:Running task workunit... 2023-07-12T15:28:43.430 INFO:tasks.workunit:Pulling workunits from ref b1bd054042b7797ba5b70d145a1da31693dda494 2023-07-12T15:28:43.430 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-07-12T15:28:43.431 DEBUG:teuthology.orchestra.run.smithi069:> stat -- /home/ubuntu/cephtest/mnt.0 2023-07-12T15:28:43.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-12T15:28:43.449 INFO:teuthology.orchestra.run.smithi069.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-07-12T15:28:43.449 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-07-12T15:28:43.506 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-07-12T15:28:43.507 DEBUG:teuthology.orchestra.run.smithi069:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-07-12T15:28:43.562 INFO:tasks.workunit:timeout=3h 2023-07-12T15:28:43.563 INFO:tasks.workunit:cleanup=True 2023-07-12T15:28:43.563 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b1bd054042b7797ba5b70d145a1da31693dda494 2023-07-12T15:28:43.620 INFO:tasks.workunit.client.0.smithi069.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-07-12T15:28:43.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1361724873' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-12T15:28:43.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:43 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:28:43.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1361724873' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-12T15:28:43.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:43 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:28:43.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1361724873' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-12T15:28:43.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:43 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:28:44.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:44 smithi069 ceph-mon[102777]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:44.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:44 smithi069 ceph-mon[94241]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:44.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:28:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:28:44.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:44 smithi145 ceph-mon[95801]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:45.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:45 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:45.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:45 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:45.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:45 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:46.465 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:28:46] "GET /metrics HTTP/1.1" 200 219498 "" "Prometheus/2.43.0" 2023-07-12T15:28:46.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:46 smithi069 ceph-mon[102777]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:46.736 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:28:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[132429]: ts=2023-07-12T15:28:46.481Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002635062s 2023-07-12T15:28:46.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:46 smithi069 ceph-mon[94241]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:46.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:46 smithi145 ceph-mon[95801]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:48.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:48 smithi069 ceph-mon[102777]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:48.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:48 smithi069 ceph-mon[94241]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:48.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:48 smithi145 ceph-mon[95801]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:50.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:50 smithi069 ceph-mon[94241]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:50.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:50 smithi069 ceph-mon[102777]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:50.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:50 smithi145 ceph-mon[95801]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:52.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:52 smithi069 ceph-mon[102777]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:52.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:52 smithi069 ceph-mon[94241]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:52.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:52 smithi145 ceph-mon[95801]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:54.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:54 smithi069 ceph-mon[102777]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:54.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:54 smithi069 ceph-mon[94241]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:54.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:28:54 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:28:54.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:54 smithi145 ceph-mon[95801]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:55.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:55 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:55.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:55 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:55.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:55 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:28:56.480 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:28:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:28:56] "GET /metrics HTTP/1.1" 200 219498 "" "Prometheus/2.43.0" 2023-07-12T15:28:56.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:56 smithi069 ceph-mon[94241]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:56.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:56 smithi069 ceph-mon[102777]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:56.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:56 smithi145 ceph-mon[95801]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:28:58.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:58 smithi069 ceph-mon[102777]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:58.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:28:58 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:28:58.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:58 smithi069 ceph-mon[94241]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:58.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:28:58 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:28:58.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:58 smithi145 ceph-mon[95801]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:28:58.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:28:58 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:29:00.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:00 smithi145 ceph-mon[95801]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:00.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:00 smithi069 ceph-mon[102777]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:00.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:00 smithi069 ceph-mon[94241]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:02.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:02 smithi145 ceph-mon[95801]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:02.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:02 smithi069 ceph-mon[102777]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:02 smithi069 ceph-mon[94241]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:04.851 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:29:04 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:29:04.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:04 smithi145 ceph-mon[95801]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:04.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:04 smithi069 ceph-mon[102777]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:04.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:04 smithi069 ceph-mon[94241]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:05.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:05 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:05.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:05 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:05.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:05 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:06.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:29:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:29:06] "GET /metrics HTTP/1.1" 200 219492 "" "Prometheus/2.43.0" 2023-07-12T15:29:06.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:06 smithi145 ceph-mon[95801]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:06.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:06 smithi069 ceph-mon[102777]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:06.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:06 smithi069 ceph-mon[94241]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:08.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:08 smithi145 ceph-mon[95801]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:08.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:08 smithi069 ceph-mon[102777]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:08.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:08 smithi069 ceph-mon[94241]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:10.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:10 smithi145 ceph-mon[95801]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:10.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:10 smithi069 ceph-mon[102777]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:10.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:10 smithi069 ceph-mon[94241]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:12.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:12 smithi145 ceph-mon[95801]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:12.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:12 smithi069 ceph-mon[102777]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:12.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:12 smithi069 ceph-mon[94241]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:13.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2023-07-12T15:29:13.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2023-07-12T15:29:13.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:13 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2023-07-12T15:29:13.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2023-07-12T15:29:13.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2023-07-12T15:29:13.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:13 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2023-07-12T15:29:13.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:13 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2023-07-12T15:29:13.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:13 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2023-07-12T15:29:13.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:29:13.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2023-07-12T15:29:13.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2023-07-12T15:29:13.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2023-07-12T15:29:13.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2023-07-12T15:29:13.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2023-07-12T15:29:13.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2023-07-12T15:29:13.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2023-07-12T15:29:13.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2023-07-12T15:29:13.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:29:13.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2023-07-12T15:29:13.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2023-07-12T15:29:13.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2023-07-12T15:29:13.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2023-07-12T15:29:13.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2023-07-12T15:29:13.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2023-07-12T15:29:13.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2023-07-12T15:29:13.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2023-07-12T15:29:13.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:29:14.851 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:29:14 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:29:14.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:14 smithi145 ceph-mon[95801]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:14.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]': finished 2023-07-12T15:29:14.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2023-07-12T15:29:14.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2023-07-12T15:29:14.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2023-07-12T15:29:14.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:14 smithi145 ceph-mon[95801]: osdmap e70: 8 total, 8 up, 8 in 2023-07-12T15:29:14.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:14 smithi069 ceph-mon[102777]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:14.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]': finished 2023-07-12T15:29:14.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2023-07-12T15:29:14.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2023-07-12T15:29:14.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2023-07-12T15:29:14.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:14 smithi069 ceph-mon[102777]: osdmap e70: 8 total, 8 up, 8 in 2023-07-12T15:29:14.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:14 smithi069 ceph-mon[94241]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:14.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]': finished 2023-07-12T15:29:14.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2023-07-12T15:29:14.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2023-07-12T15:29:14.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2023-07-12T15:29:14.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:14 smithi069 ceph-mon[94241]: osdmap e70: 8 total, 8 up, 8 in 2023-07-12T15:29:15.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:15 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:15.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:15 smithi145 ceph-mon[95801]: osdmap e71: 8 total, 8 up, 8 in 2023-07-12T15:29:15.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:15 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:15.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:15 smithi069 ceph-mon[102777]: osdmap e71: 8 total, 8 up, 8 in 2023-07-12T15:29:15.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:15 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:15.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:15 smithi069 ceph-mon[94241]: osdmap e71: 8 total, 8 up, 8 in 2023-07-12T15:29:16.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:29:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:29:16] "GET /metrics HTTP/1.1" 200 219500 "" "Prometheus/2.43.0" 2023-07-12T15:29:16.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:16 smithi145 ceph-mon[95801]: pgmap v36: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:16.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:16 smithi145 ceph-mon[95801]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2023-07-12T15:29:16.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:16 smithi069 ceph-mon[102777]: pgmap v36: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:16.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:16 smithi069 ceph-mon[102777]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2023-07-12T15:29:16.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:16 smithi069 ceph-mon[94241]: pgmap v36: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:16.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:16 smithi069 ceph-mon[94241]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2023-07-12T15:29:18.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:18 smithi145 ceph-mon[95801]: pgmap v37: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:29:18.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:18 smithi069 ceph-mon[102777]: pgmap v37: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:29:18.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:18 smithi069 ceph-mon[94241]: pgmap v37: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:29:19.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:19 smithi145 ceph-mon[95801]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2023-07-12T15:29:19.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:19 smithi145 ceph-mon[95801]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2023-07-12T15:29:19.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:19 smithi145 ceph-mon[95801]: Cluster is now healthy 2023-07-12T15:29:19.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:19 smithi069 ceph-mon[102777]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2023-07-12T15:29:19.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:19 smithi069 ceph-mon[102777]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2023-07-12T15:29:19.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:19 smithi069 ceph-mon[102777]: Cluster is now healthy 2023-07-12T15:29:19.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:19 smithi069 ceph-mon[94241]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2023-07-12T15:29:19.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:19 smithi069 ceph-mon[94241]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2023-07-12T15:29:19.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:19 smithi069 ceph-mon[94241]: Cluster is now healthy 2023-07-12T15:29:22.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:22 smithi069 ceph-mon[102777]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2023-07-12T15:29:22.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:22 smithi069 ceph-mon[94241]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2023-07-12T15:29:22.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:22 smithi145 ceph-mon[95801]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2023-07-12T15:29:24.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:24 smithi069 ceph-mon[102777]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 2 objects/s recovering 2023-07-12T15:29:24.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:24 smithi069 ceph-mon[94241]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 2 objects/s recovering 2023-07-12T15:29:24.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:29:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:29:24.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:24 smithi145 ceph-mon[95801]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 2 objects/s recovering 2023-07-12T15:29:25.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:25 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:25.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:25 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:25.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:25 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:26.443 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:29:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:29:26] "GET /metrics HTTP/1.1" 200 219500 "" "Prometheus/2.43.0" 2023-07-12T15:29:26.444 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:26 smithi069 ceph-mon[94241]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 941 B/s rd, 0 op/s; 20 B/s, 2 objects/s recovering 2023-07-12T15:29:26.444 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:26 smithi069 ceph-mon[102777]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 941 B/s rd, 0 op/s; 20 B/s, 2 objects/s recovering 2023-07-12T15:29:26.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:26 smithi145 ceph-mon[95801]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 941 B/s rd, 0 op/s; 20 B/s, 2 objects/s recovering 2023-07-12T15:29:28.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:28 smithi069 ceph-mon[102777]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2023-07-12T15:29:28.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:28 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:29:28.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:28 smithi069 ceph-mon[94241]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2023-07-12T15:29:28.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:28 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:29:28.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:28 smithi145 ceph-mon[95801]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2023-07-12T15:29:28.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:28 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:29:29.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:29 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:29 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:29.815 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:29 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:30.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:30 smithi069 ceph-mon[102777]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18 B/s, 1 objects/s recovering 2023-07-12T15:29:30.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:30 smithi069 ceph-mon[94241]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18 B/s, 1 objects/s recovering 2023-07-12T15:29:30.738 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:30 smithi145 ceph-mon[95801]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18 B/s, 1 objects/s recovering 2023-07-12T15:29:31.883 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:31 smithi145 systemd[1]: Starting Ceph grafana.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:29:32.164 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:31 smithi145 podman[122782]: 2023-07-12T15:29:32.417 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 bash[122782]: d5a713cdadad1650813e8fd7cae46e14ba85ba835ee4bf8c615a85617dc22052 2023-07-12T15:29:32.417 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 systemd[1]: Started Ceph grafana.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:29:32.417 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.298348346Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2023-07-12T15:29:32.417 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.298521145Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2023-07-12T15:29:32.418 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.298540035Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2023-07-12T15:29:32.418 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.298546458Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2023-07-12T15:29:32.418 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.298552697Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2023-07-12T15:29:32.418 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.298558457Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2023-07-12T15:29:32.418 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.298563825Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2023-07-12T15:29:32.418 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.298569322Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2023-07-12T15:29:32.418 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.298575306Z level=info msg="Path Home" path=/usr/share/grafana 2023-07-12T15:29:32.419 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.298581025Z level=info msg="Path Data" path=/var/lib/grafana 2023-07-12T15:29:32.419 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.298586134Z level=info msg="Path Logs" path=/var/log/grafana 2023-07-12T15:29:32.419 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.29859117Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2023-07-12T15:29:32.419 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.298596652Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2023-07-12T15:29:32.419 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=settings t=2023-07-12T15:29:32.298601994Z level=info msg="App mode production" 2023-07-12T15:29:32.419 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=sqlstore t=2023-07-12T15:29:32.29866456Z level=info msg="Connecting to DB" dbtype=sqlite3 2023-07-12T15:29:32.419 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=sqlstore t=2023-07-12T15:29:32.298682913Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2023-07-12T15:29:32.420 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.299538231Z level=info msg="Starting DB migrations" 2023-07-12T15:29:32.420 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.299609749Z level=info msg="Executing migration" id="create migration_log table" 2023-07-12T15:29:32.420 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.300136092Z level=info msg="Executing migration" id="create user table" 2023-07-12T15:29:32.420 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.300543871Z level=info msg="Executing migration" id="add unique index user.login" 2023-07-12T15:29:32.420 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.300948904Z level=info msg="Executing migration" id="add unique index user.email" 2023-07-12T15:29:32.420 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.301372872Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2023-07-12T15:29:32.420 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.301758319Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2023-07-12T15:29:32.421 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.302138831Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2023-07-12T15:29:32.421 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.302841178Z level=info msg="Executing migration" id="create user table v2" 2023-07-12T15:29:32.421 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.303252042Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2023-07-12T15:29:32.421 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.303600895Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2023-07-12T15:29:32.421 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.303994482Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2023-07-12T15:29:32.421 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.304329773Z level=info msg="Executing migration" id="Drop old table user_v1" 2023-07-12T15:29:32.422 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.304639024Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2023-07-12T15:29:32.422 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.305046629Z level=info msg="Executing migration" id="Update user table charset" 2023-07-12T15:29:32.422 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.305326567Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2023-07-12T15:29:32.422 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.305690809Z level=info msg="Executing migration" id="Add missing user data" 2023-07-12T15:29:32.422 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.306059833Z level=info msg="Executing migration" id="Add is_disabled column to user" 2023-07-12T15:29:32.422 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.30647654Z level=info msg="Executing migration" id="Add index user.login/user.email" 2023-07-12T15:29:32.422 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.306831799Z level=info msg="Executing migration" id="Add is_service_account column to user" 2023-07-12T15:29:32.422 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.308839681Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2023-07-12T15:29:32.423 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.309963124Z level=info msg="Executing migration" id="create temp user table v1-7" 2023-07-12T15:29:32.423 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.310376868Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2023-07-12T15:29:32.423 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.31079002Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2023-07-12T15:29:32.423 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.311171867Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2023-07-12T15:29:32.424 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.311541709Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2023-07-12T15:29:32.424 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.311914182Z level=info msg="Executing migration" id="Update temp_user table charset" 2023-07-12T15:29:32.424 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.312191095Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2023-07-12T15:29:32.425 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.312522629Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2023-07-12T15:29:32.425 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.312866538Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2023-07-12T15:29:32.425 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.313251851Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2023-07-12T15:29:32.425 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.313592854Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2023-07-12T15:29:32.426 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.314221765Z level=info msg="Executing migration" id="create temp_user v2" 2023-07-12T15:29:32.426 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.314672582Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2023-07-12T15:29:32.426 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.315113266Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2023-07-12T15:29:32.426 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.315519561Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2023-07-12T15:29:32.427 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.315911303Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2023-07-12T15:29:32.427 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.316335236Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2023-07-12T15:29:32.427 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.316656061Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2023-07-12T15:29:32.427 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.317028613Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2023-07-12T15:29:32.428 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.317374495Z level=info msg="Executing migration" id="create star table" 2023-07-12T15:29:32.428 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.31772651Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2023-07-12T15:29:32.428 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.31815752Z level=info msg="Executing migration" id="create org table v1" 2023-07-12T15:29:32.429 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.318556194Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2023-07-12T15:29:32.429 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.318967194Z level=info msg="Executing migration" id="create org_user table v1" 2023-07-12T15:29:32.429 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.319351598Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2023-07-12T15:29:32.429 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.319754208Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2023-07-12T15:29:32.430 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.320180826Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2023-07-12T15:29:32.430 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.320590966Z level=info msg="Executing migration" id="Update org table charset" 2023-07-12T15:29:32.430 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.321478364Z level=info msg="Executing migration" id="Update org_user table charset" 2023-07-12T15:29:32.430 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.322412617Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2023-07-12T15:29:32.431 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.323077588Z level=info msg="Executing migration" id="create dashboard table" 2023-07-12T15:29:32.431 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.323576206Z level=info msg="Executing migration" id="add index dashboard.account_id" 2023-07-12T15:29:32.431 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.324468125Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2023-07-12T15:29:32.432 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.325230624Z level=info msg="Executing migration" id="create dashboard_tag table" 2023-07-12T15:29:32.432 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.326027422Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2023-07-12T15:29:32.432 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.326455893Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2023-07-12T15:29:32.432 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.327397393Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2023-07-12T15:29:32.433 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.328838372Z level=info msg="Executing migration" id="create dashboard v2" 2023-07-12T15:29:32.433 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.329662467Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2023-07-12T15:29:32.435 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.330424541Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2023-07-12T15:29:32.435 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.331260126Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2023-07-12T15:29:32.435 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.332182874Z level=info msg="Executing migration" id="drop table dashboard_v1" 2023-07-12T15:29:32.435 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.333115805Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2023-07-12T15:29:32.435 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.333738103Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2023-07-12T15:29:32.435 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.334606458Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2023-07-12T15:29:32.436 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.335393417Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2023-07-12T15:29:32.436 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.336376313Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2023-07-12T15:29:32.436 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.337078441Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2023-07-12T15:29:32.436 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.338044713Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2023-07-12T15:29:32.436 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.338773788Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2023-07-12T15:29:32.436 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.339657709Z level=info msg="Executing migration" id="Update dashboard table charset" 2023-07-12T15:29:32.437 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.340628735Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2023-07-12T15:29:32.437 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.341438921Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2023-07-12T15:29:32.437 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.342109558Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2023-07-12T15:29:32.437 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.342672065Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2023-07-12T15:29:32.437 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.343513718Z level=info msg="Executing migration" id="Add column uid in dashboard" 2023-07-12T15:29:32.437 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.344413292Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2023-07-12T15:29:32.437 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.345194809Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2023-07-12T15:29:32.438 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.346101385Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2023-07-12T15:29:32.438 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.346971283Z level=info msg="Executing migration" id="Update dashboard title length" 2023-07-12T15:29:32.438 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.34774117Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2023-07-12T15:29:32.438 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.348727345Z level=info msg="Executing migration" id="create dashboard_provisioning" 2023-07-12T15:29:32.438 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.349476789Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2023-07-12T15:29:32.438 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.350421178Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2023-07-12T15:29:32.438 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.353420641Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2023-07-12T15:29:32.439 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.355817034Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2023-07-12T15:29:32.439 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.358650593Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2023-07-12T15:29:32.439 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.361224087Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2023-07-12T15:29:32.439 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.364014988Z level=info msg="Executing migration" id="Add check_sum column" 2023-07-12T15:29:32.439 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.366676572Z level=info msg="Executing migration" id="Add index for dashboard_title" 2023-07-12T15:29:32.439 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.369458656Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2023-07-12T15:29:32.439 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.372263675Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2023-07-12T15:29:32.440 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.374912966Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2023-07-12T15:29:32.440 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.377818722Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2023-07-12T15:29:32.440 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.380539173Z level=info msg="Executing migration" id="create data_source table" 2023-07-12T15:29:32.440 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.38316581Z level=info msg="Executing migration" id="add index data_source.account_id" 2023-07-12T15:29:32.440 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.385653974Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2023-07-12T15:29:32.441 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.386137835Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2023-07-12T15:29:32.441 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.389052275Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2023-07-12T15:29:32.441 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.391905167Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2023-07-12T15:29:32.441 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.394694225Z level=info msg="Executing migration" id="create data_source table v2" 2023-07-12T15:29:32.441 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.397232283Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2023-07-12T15:29:32.441 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.400093531Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2023-07-12T15:29:32.442 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.402569971Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2023-07-12T15:29:32.442 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.405194741Z level=info msg="Executing migration" id="Add column with_credentials" 2023-07-12T15:29:32.442 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.407908713Z level=info msg="Executing migration" id="Add secure json data column" 2023-07-12T15:29:32.442 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.410579243Z level=info msg="Executing migration" id="Update data_source table charset" 2023-07-12T15:29:32.443 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.413268115Z level=info msg="Executing migration" id="Update initial version to 1" 2023-07-12T15:29:32.443 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.415719967Z level=info msg="Executing migration" id="Add read_only data column" 2023-07-12T15:29:32.667 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-mon[95801]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:32.668 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:32.668 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:32.668 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:32.669 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:32.671 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.418430697Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2023-07-12T15:29:32.671 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.421165551Z level=info msg="Executing migration" id="Update json_data with nulls" 2023-07-12T15:29:32.671 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.423815302Z level=info msg="Executing migration" id="Add uid column" 2023-07-12T15:29:32.672 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.426466525Z level=info msg="Executing migration" id="Update uid value" 2023-07-12T15:29:32.672 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.429001184Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2023-07-12T15:29:32.672 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.431674742Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2023-07-12T15:29:32.672 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.434408432Z level=info msg="Executing migration" id="create api_key table" 2023-07-12T15:29:32.673 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.437086243Z level=info msg="Executing migration" id="add index api_key.account_id" 2023-07-12T15:29:32.673 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.439812851Z level=info msg="Executing migration" id="add index api_key.key" 2023-07-12T15:29:32.673 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.440252935Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2023-07-12T15:29:32.673 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.442976784Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2023-07-12T15:29:32.674 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.445714757Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2023-07-12T15:29:32.674 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.44614333Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2023-07-12T15:29:32.674 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.449006981Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2023-07-12T15:29:32.674 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.451620262Z level=info msg="Executing migration" id="create api_key table v2" 2023-07-12T15:29:32.675 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.452416662Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2023-07-12T15:29:32.675 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.452966831Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2023-07-12T15:29:32.675 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.454004177Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2023-07-12T15:29:32.675 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.454657584Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2023-07-12T15:29:32.676 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.455481513Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2023-07-12T15:29:32.676 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.455875154Z level=info msg="Executing migration" id="Update api_key table charset" 2023-07-12T15:29:32.676 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.456718014Z level=info msg="Executing migration" id="Add expires to api_key table" 2023-07-12T15:29:32.676 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.457526731Z level=info msg="Executing migration" id="Add service account foreign key" 2023-07-12T15:29:32.676 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.458507063Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2023-07-12T15:29:32.677 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.458923889Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2023-07-12T15:29:32.677 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.459719259Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2023-07-12T15:29:32.677 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.460577444Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2023-07-12T15:29:32.677 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.461482366Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2023-07-12T15:29:32.678 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.461877961Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2023-07-12T15:29:32.678 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.462736388Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2023-07-12T15:29:32.678 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.463532357Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2023-07-12T15:29:32.678 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.464505996Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2023-07-12T15:29:32.678 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.464984872Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2023-07-12T15:29:32.679 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.46581413Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2023-07-12T15:29:32.679 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.466696125Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2023-07-12T15:29:32.679 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.46762219Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2023-07-12T15:29:32.679 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.468421384Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2023-07-12T15:29:32.680 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.469201447Z level=info msg="Executing migration" id="create quota table v1" 2023-07-12T15:29:32.680 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.469962708Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2023-07-12T15:29:32.680 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.4709398Z level=info msg="Executing migration" id="Update quota table charset" 2023-07-12T15:29:32.680 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.471310842Z level=info msg="Executing migration" id="create plugin_setting table" 2023-07-12T15:29:32.681 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.472149656Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2023-07-12T15:29:32.681 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.47308781Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2023-07-12T15:29:32.681 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.473861311Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2023-07-12T15:29:32.681 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.474152704Z level=info msg="Executing migration" id="create session table" 2023-07-12T15:29:32.682 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.475049937Z level=info msg="Executing migration" id="Drop old table playlist table" 2023-07-12T15:29:32.682 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.476024394Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2023-07-12T15:29:32.682 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.476762192Z level=info msg="Executing migration" id="create playlist table v2" 2023-07-12T15:29:32.682 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.477751012Z level=info msg="Executing migration" id="create playlist item table v2" 2023-07-12T15:29:32.682 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.478509066Z level=info msg="Executing migration" id="Update playlist table charset" 2023-07-12T15:29:32.683 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.481295998Z level=info msg="Executing migration" id="Update playlist_item table charset" 2023-07-12T15:29:32.683 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.483930309Z level=info msg="Executing migration" id="drop preferences table v2" 2023-07-12T15:29:32.683 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.486756014Z level=info msg="Executing migration" id="drop preferences table v3" 2023-07-12T15:29:32.683 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.489703412Z level=info msg="Executing migration" id="create preferences table v3" 2023-07-12T15:29:32.683 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.49225725Z level=info msg="Executing migration" id="Update preferences table charset" 2023-07-12T15:29:32.683 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.492541858Z level=info msg="Executing migration" id="Add column team_id in preferences" 2023-07-12T15:29:32.683 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.495354136Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2023-07-12T15:29:32.684 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.498097362Z level=info msg="Executing migration" id="Add column week_start in preferences" 2023-07-12T15:29:32.684 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.501063512Z level=info msg="Executing migration" id="Add column preferences.json_data" 2023-07-12T15:29:32.684 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.501758389Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2023-07-12T15:29:32.684 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.504734853Z level=info msg="Executing migration" id="Add preferences index org_id" 2023-07-12T15:29:32.684 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.507562428Z level=info msg="Executing migration" id="Add preferences index user_id" 2023-07-12T15:29:32.684 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.51023743Z level=info msg="Executing migration" id="create alert table v1" 2023-07-12T15:29:32.684 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.510693414Z level=info msg="Executing migration" id="add index alert org_id & id " 2023-07-12T15:29:32.685 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.513356994Z level=info msg="Executing migration" id="add index alert state" 2023-07-12T15:29:32.685 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.516049565Z level=info msg="Executing migration" id="add index alert dashboard_id" 2023-07-12T15:29:32.685 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.519002494Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2023-07-12T15:29:32.685 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.519406355Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2023-07-12T15:29:32.685 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.522036406Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2023-07-12T15:29:32.685 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.524938436Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2023-07-12T15:29:32.685 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.527566691Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2023-07-12T15:29:32.686 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.527941857Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2023-07-12T15:29:32.686 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.530485673Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2023-07-12T15:29:32.686 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.53309706Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2023-07-12T15:29:32.686 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.535792497Z level=info msg="Executing migration" id="create alert_notification table v1" 2023-07-12T15:29:32.686 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.536196816Z level=info msg="Executing migration" id="Add column is_default" 2023-07-12T15:29:32.686 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.53867983Z level=info msg="Executing migration" id="Add column frequency" 2023-07-12T15:29:32.686 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.539345859Z level=info msg="Executing migration" id="Add column send_reminder" 2023-07-12T15:29:32.686 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.541271089Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2023-07-12T15:29:32.687 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.543987414Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2023-07-12T15:29:32.687 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.544393993Z level=info msg="Executing migration" id="Update alert table charset" 2023-07-12T15:29:32.687 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.547010971Z level=info msg="Executing migration" id="Update alert_notification table charset" 2023-07-12T15:29:32.687 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.547306364Z level=info msg="Executing migration" id="create notification_journal table v1" 2023-07-12T15:29:32.687 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.549834225Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2023-07-12T15:29:32.687 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.552358237Z level=info msg="Executing migration" id="drop alert_notification_journal" 2023-07-12T15:29:32.688 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.55272286Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2023-07-12T15:29:32.688 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.553584891Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2023-07-12T15:29:32.688 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.554497881Z level=info msg="Executing migration" id="Add for to alert table" 2023-07-12T15:29:32.688 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.555258537Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2023-07-12T15:29:32.688 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.555892842Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2023-07-12T15:29:32.688 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.556707771Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2023-07-12T15:29:32.688 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.557701989Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2023-07-12T15:29:32.688 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.558379955Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2023-07-12T15:29:32.689 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.559049287Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2023-07-12T15:29:32.689 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.559947372Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2023-07-12T15:29:32.689 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.560680961Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2023-07-12T15:29:32.689 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.561704497Z level=info msg="Executing migration" id="Drop old annotation table v4" 2023-07-12T15:29:32.689 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.562196613Z level=info msg="Executing migration" id="create annotation table v5" 2023-07-12T15:29:32.689 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.563087167Z level=info msg="Executing migration" id="add index annotation 0 v3" 2023-07-12T15:29:32.689 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.563813786Z level=info msg="Executing migration" id="add index annotation 1 v3" 2023-07-12T15:29:32.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.564782088Z level=info msg="Executing migration" id="add index annotation 2 v3" 2023-07-12T15:29:32.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.565348623Z level=info msg="Executing migration" id="add index annotation 3 v3" 2023-07-12T15:29:32.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.566164549Z level=info msg="Executing migration" id="add index annotation 4 v3" 2023-07-12T15:29:32.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.567011627Z level=info msg="Executing migration" id="Update annotation table charset" 2023-07-12T15:29:32.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.567892464Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2023-07-12T15:29:32.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.56852896Z level=info msg="Executing migration" id="Drop category_id index" 2023-07-12T15:29:32.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.569370996Z level=info msg="Executing migration" id="Add column tags to annotation table" 2023-07-12T15:29:32.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.570037141Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2023-07-12T15:29:32.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.570431623Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2023-07-12T15:29:32.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.571305687Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2023-07-12T15:29:32.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.571723386Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2023-07-12T15:29:32.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.573435108Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2023-07-12T15:29:32.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.574416675Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2023-07-12T15:29:32.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.575138726Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2023-07-12T15:29:32.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.575475928Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2023-07-12T15:29:32.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.576355125Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2023-07-12T15:29:32.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.576690774Z level=info msg="Executing migration" id="Add created time to annotation table" 2023-07-12T15:29:32.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.577421489Z level=info msg="Executing migration" id="Add updated time to annotation table" 2023-07-12T15:29:32.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.578248519Z level=info msg="Executing migration" id="Add index for created in annotation table" 2023-07-12T15:29:32.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.578696012Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2023-07-12T15:29:32.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.581246144Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2023-07-12T15:29:32.693 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.584056828Z level=info msg="Executing migration" id="Add epoch_end column" 2023-07-12T15:29:32.693 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.586860705Z level=info msg="Executing migration" id="Add index for epoch_end" 2023-07-12T15:29:32.693 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.587345621Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2023-07-12T15:29:32.693 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.590074918Z level=info msg="Executing migration" id="Move region to single row" 2023-07-12T15:29:32.693 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.590444532Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2023-07-12T15:29:32.693 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.592689632Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2023-07-12T15:29:32.693 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.593118156Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2023-07-12T15:29:32.694 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.595511254Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2023-07-12T15:29:32.694 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.595936618Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2023-07-12T15:29:32.694 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.59863524Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2023-07-12T15:29:32.694 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.601591553Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2023-07-12T15:29:32.694 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.604135298Z level=info msg="Executing migration" id="create test_data table" 2023-07-12T15:29:32.694 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.604543043Z level=info msg="Executing migration" id="create dashboard_version table v1" 2023-07-12T15:29:32.694 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.604924082Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2023-07-12T15:29:32.695 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.605788213Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2023-07-12T15:29:32.695 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.606621994Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2023-07-12T15:29:32.695 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.606927578Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2023-07-12T15:29:32.695 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.607901666Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2023-07-12T15:29:32.695 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.608571812Z level=info msg="Executing migration" id="create team table" 2023-07-12T15:29:32.695 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.609610971Z level=info msg="Executing migration" id="add index team.org_id" 2023-07-12T15:29:32.695 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.610161702Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2023-07-12T15:29:32.695 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.610986703Z level=info msg="Executing migration" id="create team member table" 2023-07-12T15:29:32.696 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.611911318Z level=info msg="Executing migration" id="add index team_member.org_id" 2023-07-12T15:29:32.696 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.612783268Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2023-07-12T15:29:32.696 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.613362498Z level=info msg="Executing migration" id="add index team_member.team_id" 2023-07-12T15:29:32.696 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.614248332Z level=info msg="Executing migration" id="Add column email to team table" 2023-07-12T15:29:32.696 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.61495923Z level=info msg="Executing migration" id="Add column external to team_member table" 2023-07-12T15:29:32.696 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.61573524Z level=info msg="Executing migration" id="Add column permission to team_member table" 2023-07-12T15:29:32.696 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.616455314Z level=info msg="Executing migration" id="create dashboard acl table" 2023-07-12T15:29:32.697 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.617354759Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2023-07-12T15:29:32.697 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.618149066Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2023-07-12T15:29:32.697 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.618553626Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2023-07-12T15:29:32.697 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.618968284Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2023-07-12T15:29:32.697 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.619842404Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2023-07-12T15:29:32.697 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.620804537Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2023-07-12T15:29:32.697 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.621534181Z level=info msg="Executing migration" id="add index dashboard_permission" 2023-07-12T15:29:32.697 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.622481847Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2023-07-12T15:29:32.698 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.623282449Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2023-07-12T15:29:32.698 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.624225034Z level=info msg="Executing migration" id="create tag table" 2023-07-12T15:29:32.698 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.624984287Z level=info msg="Executing migration" id="add index tag.key_value" 2023-07-12T15:29:32.698 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.625854911Z level=info msg="Executing migration" id="create login attempt table" 2023-07-12T15:29:32.698 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.626887737Z level=info msg="Executing migration" id="add index login_attempt.username" 2023-07-12T15:29:32.698 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.627819786Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2023-07-12T15:29:32.698 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.628726071Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2023-07-12T15:29:32.699 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.630709644Z level=info msg="Executing migration" id="create login_attempt v2" 2023-07-12T15:29:32.699 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.631581624Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2023-07-12T15:29:32.699 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.634389615Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2023-07-12T15:29:32.699 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.636888517Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2023-07-12T15:29:32.699 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.639629374Z level=info msg="Executing migration" id="create user auth table" 2023-07-12T15:29:32.699 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.642271676Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2023-07-12T15:29:32.699 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.645079339Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2023-07-12T15:29:32.699 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.647747907Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2023-07-12T15:29:32.700 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.650652077Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2023-07-12T15:29:32.700 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.653220475Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2023-07-12T15:29:32.700 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.655904428Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2023-07-12T15:29:32.700 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.658966466Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2023-07-12T15:29:32.700 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.661515742Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2023-07-12T15:29:32.700 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.664204801Z level=info msg="Executing migration" id="create server_lock table" 2023-07-12T15:29:32.700 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.666735769Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2023-07-12T15:29:32.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:32 smithi069 ceph-mon[102777]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:32 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:32 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:32 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:32 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:32.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:32 smithi069 ceph-mon[94241]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:32.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:32 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:32.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:32 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:32.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:32 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:32.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:32 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:32.920 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.669635205Z level=info msg="Executing migration" id="create user auth token table" 2023-07-12T15:29:32.920 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.672372886Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2023-07-12T15:29:32.921 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.674983301Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2023-07-12T15:29:32.921 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.677673602Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2023-07-12T15:29:32.921 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.680429545Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2023-07-12T15:29:32.921 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.683056297Z level=info msg="Executing migration" id="create cache_data table" 2023-07-12T15:29:32.921 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.685596575Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2023-07-12T15:29:32.921 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.688069867Z level=info msg="Executing migration" id="create short_url table v1" 2023-07-12T15:29:32.921 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.690599485Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2023-07-12T15:29:32.922 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.693397439Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2023-07-12T15:29:32.922 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.695895351Z level=info msg="Executing migration" id="delete alert_definition table" 2023-07-12T15:29:32.922 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.696211448Z level=info msg="Executing migration" id="recreate alert_definition table" 2023-07-12T15:29:32.922 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.696669229Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2023-07-12T15:29:32.922 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.699363459Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2023-07-12T15:29:32.922 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.699968863Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2023-07-12T15:29:32.922 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.70026633Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2023-07-12T15:29:32.922 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.700659601Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2023-07-12T15:29:32.923 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.701096913Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2023-07-12T15:29:32.923 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.701555857Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2023-07-12T15:29:32.923 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.701985577Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2023-07-12T15:29:32.923 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.702779726Z level=info msg="Executing migration" id="drop alert_definition table" 2023-07-12T15:29:32.923 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.703191103Z level=info msg="Executing migration" id="delete alert_definition_version table" 2023-07-12T15:29:32.923 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.703489309Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2023-07-12T15:29:32.923 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.703875768Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2023-07-12T15:29:32.924 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.704330744Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2023-07-12T15:29:32.924 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.704743658Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2023-07-12T15:29:32.924 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.705064427Z level=info msg="Executing migration" id="drop alert_definition_version table" 2023-07-12T15:29:32.924 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.705495248Z level=info msg="Executing migration" id="create alert_instance table" 2023-07-12T15:29:32.924 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.705877837Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2023-07-12T15:29:32.924 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.706324618Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2023-07-12T15:29:32.924 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.706737874Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2023-07-12T15:29:32.925 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.707550289Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2023-07-12T15:29:32.925 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.707963915Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2023-07-12T15:29:32.925 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.70840132Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2023-07-12T15:29:32.925 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.711968987Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2023-07-12T15:29:32.925 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.715364367Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2023-07-12T15:29:32.925 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.715820802Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2023-07-12T15:29:32.925 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.716272384Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2023-07-12T15:29:32.925 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.717094838Z level=info msg="Executing migration" id="create alert_rule table" 2023-07-12T15:29:32.926 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.717497466Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2023-07-12T15:29:32.926 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.717932962Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2023-07-12T15:29:32.926 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.71836413Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2023-07-12T15:29:32.926 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.718782329Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2023-07-12T15:29:32.926 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.719083604Z level=info msg="Executing migration" id="add column for to alert_rule" 2023-07-12T15:29:32.926 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.719904451Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2023-07-12T15:29:32.926 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.720711565Z level=info msg="Executing migration" id="add column labels to alert_rule" 2023-07-12T15:29:32.927 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.7215428Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2023-07-12T15:29:32.927 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.721966456Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2023-07-12T15:29:32.927 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.72241284Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2023-07-12T15:29:32.927 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.723263638Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2023-07-12T15:29:32.927 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.724093504Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2023-07-12T15:29:32.927 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.724527869Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2023-07-12T15:29:32.927 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.725404451Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2023-07-12T15:29:32.927 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.726255631Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2023-07-12T15:29:32.928 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.726567386Z level=info msg="Executing migration" id="create alert_rule_version table" 2023-07-12T15:29:32.929 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.727024976Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2023-07-12T15:29:32.929 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.727481369Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2023-07-12T15:29:32.929 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.727915062Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2023-07-12T15:29:32.929 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.72984166Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2023-07-12T15:29:32.929 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.732383916Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2023-07-12T15:29:32.929 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.735118617Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2023-07-12T15:29:32.929 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.737763797Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2023-07-12T15:29:32.930 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.740096907Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2023-07-12T15:29:32.930 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.743011355Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2023-07-12T15:29:32.930 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.745799759Z level=info msg="Executing migration" id=create_alert_configuration_table 2023-07-12T15:29:32.930 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.74839538Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2023-07-12T15:29:32.930 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.751025361Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2023-07-12T15:29:32.930 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.753665729Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2023-07-12T15:29:32.930 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.754580852Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2023-07-12T15:29:32.931 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.75538203Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2023-07-12T15:29:32.931 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.756294497Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2023-07-12T15:29:32.931 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.757201757Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2023-07-12T15:29:32.931 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.757943523Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2023-07-12T15:29:32.931 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.758837906Z level=info msg="Executing migration" id="create provenance_type table" 2023-07-12T15:29:32.931 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.759664938Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2023-07-12T15:29:32.932 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.760490583Z level=info msg="Executing migration" id="create alert_image table" 2023-07-12T15:29:32.932 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.761379236Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2023-07-12T15:29:32.932 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.76207794Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2023-07-12T15:29:32.932 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.762927152Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2023-07-12T15:29:32.932 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.76366349Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2023-07-12T15:29:32.932 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.76412503Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2023-07-12T15:29:32.932 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.764225211Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2023-07-12T15:29:32.932 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.765085775Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2023-07-12T15:29:32.933 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.765794668Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2023-07-12T15:29:32.933 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.766627632Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2023-07-12T15:29:32.933 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.766771514Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2023-07-12T15:29:32.933 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.766917437Z level=info msg="alerts found to migrate" alerts=0 2023-07-12T15:29:32.933 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.767437377Z level=info msg="Executing migration" id="create library_element table v1" 2023-07-12T15:29:32.933 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.769976076Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2023-07-12T15:29:32.933 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.772922244Z level=info msg="Executing migration" id="create library_element_connection table v1" 2023-07-12T15:29:32.934 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.775642799Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2023-07-12T15:29:32.934 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.778385253Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2023-07-12T15:29:32.934 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.781104077Z level=info msg="Executing migration" id="increase max description length to 2048" 2023-07-12T15:29:32.934 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.783808693Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2023-07-12T15:29:32.934 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.786589226Z level=info msg="Executing migration" id="create data_keys table" 2023-07-12T15:29:32.934 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.789239179Z level=info msg="Executing migration" id="create secrets table" 2023-07-12T15:29:32.934 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.792169551Z level=info msg="Executing migration" id="rename data_keys name column to id" 2023-07-12T15:29:32.934 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.796571027Z level=info msg="Executing migration" id="add name column into data_keys" 2023-07-12T15:29:32.935 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.799446704Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2023-07-12T15:29:32.935 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.802258446Z level=info msg="Executing migration" id="rename data_keys name column to label" 2023-07-12T15:29:32.935 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.806490861Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2023-07-12T15:29:32.935 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.810683402Z level=info msg="Executing migration" id="create kv_store table v1" 2023-07-12T15:29:32.935 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.813430217Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2023-07-12T15:29:32.935 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.816083464Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2023-07-12T15:29:32.935 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.818862023Z level=info msg="Executing migration" id="create permission table" 2023-07-12T15:29:32.936 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.821157995Z level=info msg="Executing migration" id="add unique index permission.role_id" 2023-07-12T15:29:32.936 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.823909038Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2023-07-12T15:29:32.936 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.826491105Z level=info msg="Executing migration" id="create role table" 2023-07-12T15:29:32.936 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.829127908Z level=info msg="Executing migration" id="add column display_name" 2023-07-12T15:29:32.936 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.831931032Z level=info msg="Executing migration" id="add column group_name" 2023-07-12T15:29:32.936 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.834405416Z level=info msg="Executing migration" id="add index role.org_id" 2023-07-12T15:29:32.936 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.837109433Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2023-07-12T15:29:32.937 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.839818709Z level=info msg="Executing migration" id="add index role_org_id_uid" 2023-07-12T15:29:32.937 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.842415286Z level=info msg="Executing migration" id="create team role table" 2023-07-12T15:29:32.937 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.845142594Z level=info msg="Executing migration" id="add index team_role.org_id" 2023-07-12T15:29:32.937 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.847655557Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2023-07-12T15:29:32.937 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.850437624Z level=info msg="Executing migration" id="add index team_role.team_id" 2023-07-12T15:29:32.937 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.853149956Z level=info msg="Executing migration" id="create user role table" 2023-07-12T15:29:32.937 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.853811248Z level=info msg="Executing migration" id="add index user_role.org_id" 2023-07-12T15:29:32.937 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.854790537Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2023-07-12T15:29:32.938 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.855478843Z level=info msg="Executing migration" id="add index user_role.user_id" 2023-07-12T15:29:32.938 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.856465623Z level=info msg="Executing migration" id="create builtin role table" 2023-07-12T15:29:32.938 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.857099221Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2023-07-12T15:29:32.938 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.858006055Z level=info msg="Executing migration" id="add index builtin_role.name" 2023-07-12T15:29:32.938 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.858664982Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2023-07-12T15:29:32.938 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.859680883Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2023-07-12T15:29:32.938 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.860493206Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2023-07-12T15:29:32.939 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.86141942Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2023-07-12T15:29:32.939 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.862083884Z level=info msg="Executing migration" id="add unique index role.uid" 2023-07-12T15:29:32.939 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.862948535Z level=info msg="Executing migration" id="create seed assignment table" 2023-07-12T15:29:32.939 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.863923916Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2023-07-12T15:29:32.939 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.864591313Z level=info msg="Executing migration" id="add column hidden to role table" 2023-07-12T15:29:32.939 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.865574976Z level=info msg="Executing migration" id="create query_history table v1" 2023-07-12T15:29:32.939 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.86663955Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2023-07-12T15:29:32.939 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.867314755Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2023-07-12T15:29:32.940 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.868110579Z level=info msg="Executing migration" id="teams permissions migration" 2023-07-12T15:29:32.940 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.868941379Z level=info msg="Executing migration" id="dashboard permissions" 2023-07-12T15:29:32.940 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.869954695Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2023-07-12T15:29:32.940 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.870693722Z level=info msg="Executing migration" id="drop managed folder create actions" 2023-07-12T15:29:32.940 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.871480653Z level=info msg="Executing migration" id="alerting notification permissions" 2023-07-12T15:29:32.940 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.872326965Z level=info msg="Executing migration" id="create query_history_star table v1" 2023-07-12T15:29:32.940 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.873282898Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2023-07-12T15:29:32.941 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.873996817Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2023-07-12T15:29:32.941 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.874963735Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2023-07-12T15:29:32.941 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.875945275Z level=info msg="Executing migration" id="create correlation table v1" 2023-07-12T15:29:32.941 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.876650068Z level=info msg="Executing migration" id="add index correlations.uid" 2023-07-12T15:29:32.941 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.877638248Z level=info msg="Executing migration" id="add index correlations.source_uid" 2023-07-12T15:29:32.941 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.878356474Z level=info msg="Executing migration" id="add correlation config column" 2023-07-12T15:29:32.941 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.879430434Z level=info msg="Executing migration" id="create entity_events table" 2023-07-12T15:29:32.941 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.880457257Z level=info msg="Executing migration" id="create dashboard public config v1" 2023-07-12T15:29:32.942 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.881128141Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2023-07-12T15:29:32.942 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.881252798Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2023-07-12T15:29:32.942 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.882080072Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2023-07-12T15:29:32.942 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.882265515Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2023-07-12T15:29:32.942 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.882807702Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2023-07-12T15:29:32.942 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.885565876Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2023-07-12T15:29:32.942 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.888494933Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2023-07-12T15:29:32.943 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.891245538Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2023-07-12T15:29:32.943 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.893975081Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2023-07-12T15:29:32.943 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.896781328Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2023-07-12T15:29:32.943 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.89926627Z level=info msg="Executing migration" id="Drop public config table" 2023-07-12T15:29:32.943 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.90193685Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2023-07-12T15:29:32.943 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.904717592Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2023-07-12T15:29:32.943 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.907580789Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2023-07-12T15:29:32.943 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.910082967Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2023-07-12T15:29:32.944 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.913073382Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2023-07-12T15:29:32.944 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.916262331Z level=info msg="Executing migration" id="add annotations_enabled column" 2023-07-12T15:29:32.944 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.918962581Z level=info msg="Executing migration" id="add time_selection_enabled column" 2023-07-12T15:29:32.944 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.921605565Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2023-07-12T15:29:33.158 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:33 smithi069 systemd[1]: Starting Ceph node-exporter.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:29:33.351 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.924215835Z level=info msg="Executing migration" id="add share column" 2023-07-12T15:29:33.351 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.927033099Z level=info msg="Executing migration" id="create default alerting folders" 2023-07-12T15:29:33.351 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.930060585Z level=info msg="Executing migration" id="create file table" 2023-07-12T15:29:33.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.93243935Z level=info msg="Executing migration" id="file table idx: path natural pk" 2023-07-12T15:29:33.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.935089835Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2023-07-12T15:29:33.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.937926537Z level=info msg="Executing migration" id="create file_meta table" 2023-07-12T15:29:33.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.940628228Z level=info msg="Executing migration" id="file table idx: path key" 2023-07-12T15:29:33.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.943227632Z level=info msg="Executing migration" id="set path collation in file table" 2023-07-12T15:29:33.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.945698474Z level=info msg="Executing migration" id="managed permissions migration" 2023-07-12T15:29:33.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.948259924Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2023-07-12T15:29:33.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.950991567Z level=info msg="Executing migration" id="RBAC action name migrator" 2023-07-12T15:29:33.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.953560288Z level=info msg="Executing migration" id="Add UID column to playlist" 2023-07-12T15:29:33.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.956053399Z level=info msg="Executing migration" id="Update uid column values in playlist" 2023-07-12T15:29:33.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.958666516Z level=info msg="Executing migration" id="Add index for uid in playlist" 2023-07-12T15:29:33.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.961412394Z level=info msg="Executing migration" id="update group index for alert rules" 2023-07-12T15:29:33.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.964026389Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2023-07-12T15:29:33.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.966787123Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2023-07-12T15:29:33.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.969313002Z level=info msg="Executing migration" id="add action column to seed_assignment" 2023-07-12T15:29:33.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.971931497Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2023-07-12T15:29:33.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.974599614Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2023-07-12T15:29:33.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.977270783Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2023-07-12T15:29:33.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.986941267Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2023-07-12T15:29:33.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.987799679Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2023-07-12T15:29:33.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.988619773Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2023-07-12T15:29:33.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.992002513Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2023-07-12T15:29:33.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.992745768Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2023-07-12T15:29:33.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.99356429Z level=info msg="Executing migration" id="create folder table" 2023-07-12T15:29:33.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.994436974Z level=info msg="Executing migration" id="Add index for parent_uid" 2023-07-12T15:29:33.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.995310159Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2023-07-12T15:29:33.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.996084438Z level=info msg="Executing migration" id="Update folder title length" 2023-07-12T15:29:33.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.996990027Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2023-07-12T15:29:33.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=migrator t=2023-07-12T15:29:32.997647877Z level=info msg="migrations completed" performed=485 skipped=0 duration=698.05747ms 2023-07-12T15:29:33.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=sqlstore t=2023-07-12T15:29:32.998106162Z level=info msg="Created default organization" 2023-07-12T15:29:33.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=plugin.loader t=2023-07-12T15:29:33.054248653Z level=info msg="Plugin registered" pluginID=input 2023-07-12T15:29:33.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=plugin.loader t=2023-07-12T15:29:33.082486029Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2023-07-12T15:29:33.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=plugin.loader t=2023-07-12T15:29:33.082516892Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2023-07-12T15:29:33.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=secrets t=2023-07-12T15:29:33.082549869Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2023-07-12T15:29:33.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=query_data t=2023-07-12T15:29:33.084307923Z level=info msg="Query Service initialization" 2023-07-12T15:29:33.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=live.push_http t=2023-07-12T15:29:33.086551637Z level=info msg="Live Push Gateway initialization" 2023-07-12T15:29:33.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:33 smithi069 ceph-mon[102777]: Deploying daemon node-exporter.a on smithi069 2023-07-12T15:29:33.736 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:33 smithi069 bash[133395]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2023-07-12T15:29:33.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:33 smithi069 ceph-mon[94241]: Deploying daemon node-exporter.a on smithi069 2023-07-12T15:29:33.790 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-mon[95801]: Deploying daemon node-exporter.a on smithi069 2023-07-12T15:29:34.100 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=infra.usagestats.collector t=2023-07-12T15:29:33.789602906Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2023-07-12T15:29:34.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=provisioning.datasources t=2023-07-12T15:29:33.842968267Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2023-07-12T15:29:34.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=provisioning.datasources t=2023-07-12T15:29:33.874672801Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2023-07-12T15:29:34.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=provisioning.alerting t=2023-07-12T15:29:33.896225827Z level=info msg="starting to provision alerting" 2023-07-12T15:29:34.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=provisioning.alerting t=2023-07-12T15:29:33.896251287Z level=info msg="finished to provision alerting" 2023-07-12T15:29:34.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=grafanaStorageLogger t=2023-07-12T15:29:33.896398547Z level=info msg="storage starting" 2023-07-12T15:29:34.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=ngalert.state.manager t=2023-07-12T15:29:33.896515916Z level=info msg="Warming state cache for startup" 2023-07-12T15:29:34.102 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=http.server t=2023-07-12T15:29:33.898076025Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2023-07-12T15:29:34.102 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=sqlstore.transactions t=2023-07-12T15:29:33.907225977Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2023-07-12T15:29:34.102 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=ngalert.state.manager t=2023-07-12T15:29:33.907471628Z level=info msg="State cache has been initialized" states=0 duration=10.952867ms 2023-07-12T15:29:34.102 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=ticker t=2023-07-12T15:29:33.907549575Z level=info msg=starting first_tick=2023-07-12T15:29:40Z 2023-07-12T15:29:34.102 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=ngalert.multiorg.alertmanager t=2023-07-12T15:29:33.907570825Z level=info msg="starting MultiOrg Alertmanager" 2023-07-12T15:29:34.102 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:29:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=sqlstore.transactions t=2023-07-12T15:29:33.942188024Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2023-07-12T15:29:34.235 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:33 smithi069 bash[133395]: Getting image source signatures 2023-07-12T15:29:34.236 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:33 smithi069 bash[133395]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2023-07-12T15:29:34.236 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:33 smithi069 bash[133395]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2023-07-12T15:29:34.236 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:33 smithi069 bash[133395]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2023-07-12T15:29:34.236 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:33 smithi069 bash[133395]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2023-07-12T15:29:34.236 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:33 smithi069 bash[133395]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2023-07-12T15:29:34.236 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:33 smithi069 bash[133395]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2023-07-12T15:29:34.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:29:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:29:34.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:34 smithi145 ceph-mon[95801]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:34.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:34 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:34.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:34 smithi069 ceph-mon[102777]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:34.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:34 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:34.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:34 smithi069 ceph-mon[94241]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:34.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:34 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:35.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:35 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:35.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:35 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:35.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:35 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:36.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:29:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:29:36] "GET /metrics HTTP/1.1" 200 219566 "" "Prometheus/2.43.0" 2023-07-12T15:29:36.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:36 smithi145 ceph-mon[95801]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:36.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:36 smithi069 ceph-mon[102777]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:36.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:36 smithi069 ceph-mon[94241]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:38.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:38 smithi069 ceph-mon[102777]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:38.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:38 smithi069 ceph-mon[94241]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:38.736 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:38 smithi069 bash[133395]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2023-07-12T15:29:38.736 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:38 smithi069 bash[133395]: Writing manifest to image destination 2023-07-12T15:29:38.736 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:38 smithi069 bash[133395]: Storing signatures 2023-07-12T15:29:38.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:38 smithi145 ceph-mon[95801]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:39.299 INFO:tasks.workunit.client.0.smithi069.stderr:Updating files: 97% (11457/11710) Updating files: 98% (11476/11710) Updating files: 99% (11593/11710) Updating files: 100% (11710/11710) Updating files: 100% (11710/11710), done. 2023-07-12T15:29:39.304 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 podman[133395]: 2023-07-12T15:29:39.736 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.303Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2023-07-12T15:29:39.736 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.303Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2023-07-12T15:29:39.736 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.304Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2023-07-12T15:29:39.736 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.304Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2023-07-12T15:29:39.737 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.304Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2023-07-12T15:29:39.737 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.304Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2023-07-12T15:29:39.737 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2023-07-12T15:29:39.737 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=arp 2023-07-12T15:29:39.737 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=bcache 2023-07-12T15:29:39.737 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=bonding 2023-07-12T15:29:39.737 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=btrfs 2023-07-12T15:29:39.738 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=conntrack 2023-07-12T15:29:39.738 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=cpu 2023-07-12T15:29:39.738 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=cpufreq 2023-07-12T15:29:39.738 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=diskstats 2023-07-12T15:29:39.738 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=dmi 2023-07-12T15:29:39.738 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=edac 2023-07-12T15:29:39.739 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=entropy 2023-07-12T15:29:39.739 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=fibrechannel 2023-07-12T15:29:39.739 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=filefd 2023-07-12T15:29:39.739 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=filesystem 2023-07-12T15:29:39.739 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=hwmon 2023-07-12T15:29:39.739 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=infiniband 2023-07-12T15:29:39.739 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=ipvs 2023-07-12T15:29:39.739 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=loadavg 2023-07-12T15:29:39.740 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=mdadm 2023-07-12T15:29:39.740 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=meminfo 2023-07-12T15:29:39.740 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=netclass 2023-07-12T15:29:39.740 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=netdev 2023-07-12T15:29:39.740 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=netstat 2023-07-12T15:29:39.740 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=nfs 2023-07-12T15:29:39.740 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=nfsd 2023-07-12T15:29:39.741 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=nvme 2023-07-12T15:29:39.741 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=os 2023-07-12T15:29:39.741 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2023-07-12T15:29:39.741 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=pressure 2023-07-12T15:29:39.741 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=rapl 2023-07-12T15:29:39.741 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=schedstat 2023-07-12T15:29:39.741 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=selinux 2023-07-12T15:29:39.742 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=sockstat 2023-07-12T15:29:39.742 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=softnet 2023-07-12T15:29:39.742 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=stat 2023-07-12T15:29:39.742 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=tapestats 2023-07-12T15:29:39.742 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=textfile 2023-07-12T15:29:39.742 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=thermal_zone 2023-07-12T15:29:39.742 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=time 2023-07-12T15:29:39.743 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=udp_queues 2023-07-12T15:29:39.743 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=uname 2023-07-12T15:29:39.743 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=vmstat 2023-07-12T15:29:39.743 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=xfs 2023-07-12T15:29:39.743 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=node_exporter.go:117 level=info collector=zfs 2023-07-12T15:29:39.744 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2023-07-12T15:29:39.744 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a[133445]: ts=2023-07-12T15:29:39.305Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2023-07-12T15:29:39.744 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 bash[133395]: 5c7b6bb02d43b53e68acf61007c86f8e57bb5e998eb1e5ab7dfc6a21bbc15ff1 2023-07-12T15:29:39.744 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:29:39 smithi069 systemd[1]: Started Ceph node-exporter.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:29:39.762 INFO:tasks.workunit.client.0.smithi069.stderr:Note: switching to 'b1bd054042b7797ba5b70d145a1da31693dda494'. 2023-07-12T15:29:39.762 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-07-12T15:29:39.763 INFO:tasks.workunit.client.0.smithi069.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-07-12T15:29:39.763 INFO:tasks.workunit.client.0.smithi069.stderr:changes and commit them, and you can discard any commits you make in this 2023-07-12T15:29:39.763 INFO:tasks.workunit.client.0.smithi069.stderr:state without impacting any branches by switching back to a branch. 2023-07-12T15:29:39.763 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-07-12T15:29:39.763 INFO:tasks.workunit.client.0.smithi069.stderr:If you want to create a new branch to retain commits you create, you may 2023-07-12T15:29:39.763 INFO:tasks.workunit.client.0.smithi069.stderr:do so (now or later) by using -c with the switch command. Example: 2023-07-12T15:29:39.763 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-07-12T15:29:39.764 INFO:tasks.workunit.client.0.smithi069.stderr: git switch -c 2023-07-12T15:29:39.764 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-07-12T15:29:39.764 INFO:tasks.workunit.client.0.smithi069.stderr:Or undo this operation with: 2023-07-12T15:29:39.764 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-07-12T15:29:39.764 INFO:tasks.workunit.client.0.smithi069.stderr: git switch - 2023-07-12T15:29:39.764 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-07-12T15:29:39.764 INFO:tasks.workunit.client.0.smithi069.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-07-12T15:29:39.765 INFO:tasks.workunit.client.0.smithi069.stderr: 2023-07-12T15:29:39.765 INFO:tasks.workunit.client.0.smithi069.stderr:HEAD is now at b1bd054042b Merge branch 'wip-61339-reef' of https://github.com/guits/ceph into wip-guits-testing-4-2023-07-12-0746 2023-07-12T15:29:39.767 DEBUG:teuthology.orchestra.run.smithi069:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-07-12T15:29:39.835 INFO:tasks.workunit.client.0.smithi069.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-07-12T15:29:39.836 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-07-12T15:29:39.837 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-07-12T15:29:40.351 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:40 smithi145 systemd[1]: Starting Ceph node-exporter.b for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:29:40.369 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-07-12T15:29:40.420 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-07-12T15:29:40.456 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-07-12T15:29:40.458 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-07-12T15:29:40.458 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-07-12T15:29:40.495 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-07-12T15:29:40.498 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-07-12T15:29:40.499 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-07-12T15:29:40.556 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-07-12T15:29:40.558 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-07-12T15:29:40.558 DEBUG:teuthology.orchestra.run.smithi069:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b1bd054042b7797ba5b70d145a1da31693dda494 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-07-12T15:29:40.629 INFO:tasks.workunit.client.0.smithi069.stderr:+ parallel=1 2023-07-12T15:29:40.630 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' '' = --serial ']' 2023-07-12T15:29:40.630 INFO:tasks.workunit.client.0.smithi069.stderr:+ color= 2023-07-12T15:29:40.630 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -t 1 ']' 2023-07-12T15:29:40.630 INFO:tasks.workunit.client.0.smithi069.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-07-12T15:29:40.630 INFO:tasks.workunit.client.0.smithi069.stderr:+ declare -A pids 2023-07-12T15:29:40.630 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.630 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.631 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_aio 2023-07-12T15:29:40.631 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_aio' 2023-07-12T15:29:40.631 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_aio 2023-07-12T15:29:40.631 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.632 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_aio 2023-07-12T15:29:40.632 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133614 2023-07-12T15:29:40.632 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_aio on pid 133614' 2023-07-12T15:29:40.632 INFO:tasks.workunit.client.0.smithi069.stdout:test api_aio on pid 133614 2023-07-12T15:29:40.633 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133614 2023-07-12T15:29:40.634 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.634 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.634 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-07-12T15:29:40.634 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_aio_pp 2023-07-12T15:29:40.634 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_aio_pp' 2023-07-12T15:29:40.634 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_aio_pp 2023-07-12T15:29:40.635 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.635 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_aio 2023-07-12T15:29:40.635 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_aio.log 2023-07-12T15:29:40.635 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_aio: /' 2023-07-12T15:29:40.635 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_aio_pp 2023-07-12T15:29:40.635 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133622 2023-07-12T15:29:40.635 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_aio_pp on pid 133622' 2023-07-12T15:29:40.636 INFO:tasks.workunit.client.0.smithi069.stdout:test api_aio_pp on pid 133622 2023-07-12T15:29:40.637 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133622 2023-07-12T15:29:40.637 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.637 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.637 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-07-12T15:29:40.637 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_io 2023-07-12T15:29:40.638 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_io' 2023-07-12T15:29:40.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_io 2023-07-12T15:29:40.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.638 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_aio_pp 2023-07-12T15:29:40.638 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-07-12T15:29:40.638 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_aio_pp: /' 2023-07-12T15:29:40.638 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_io 2023-07-12T15:29:40.639 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133630 2023-07-12T15:29:40.639 INFO:tasks.workunit.client.0.smithi069.stdout:test api_io on pid 133630 2023-07-12T15:29:40.640 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_io on pid 133630' 2023-07-12T15:29:40.640 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133630 2023-07-12T15:29:40.640 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.640 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.641 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-07-12T15:29:40.641 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_io_pp 2023-07-12T15:29:40.641 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_io_pp' 2023-07-12T15:29:40.641 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_io_pp 2023-07-12T15:29:40.641 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.641 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_io 2023-07-12T15:29:40.641 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_io.log 2023-07-12T15:29:40.642 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_io: /' 2023-07-12T15:29:40.642 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_io_pp 2023-07-12T15:29:40.642 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133638 2023-07-12T15:29:40.642 INFO:tasks.workunit.client.0.smithi069.stdout:test api_io_pp on pid 133638 2023-07-12T15:29:40.643 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_io_pp on pid 133638' 2023-07-12T15:29:40.643 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133638 2023-07-12T15:29:40.644 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.644 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.644 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-07-12T15:29:40.644 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_asio 2023-07-12T15:29:40.644 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_asio' 2023-07-12T15:29:40.644 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_asio 2023-07-12T15:29:40.644 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.645 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_io_pp 2023-07-12T15:29:40.645 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_io_pp: /' 2023-07-12T15:29:40.645 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-07-12T15:29:40.645 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_asio 2023-07-12T15:29:40.645 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133646 2023-07-12T15:29:40.645 INFO:tasks.workunit.client.0.smithi069.stdout:test api_asio on pid 133646 2023-07-12T15:29:40.647 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_asio on pid 133646' 2023-07-12T15:29:40.647 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133646 2023-07-12T15:29:40.647 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.647 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.647 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-07-12T15:29:40.647 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_list 2023-07-12T15:29:40.648 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_list' 2023-07-12T15:29:40.648 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_list 2023-07-12T15:29:40.648 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.648 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_asio 2023-07-12T15:29:40.648 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_asio: /' 2023-07-12T15:29:40.648 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_asio.log 2023-07-12T15:29:40.648 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_list 2023-07-12T15:29:40.649 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133654 2023-07-12T15:29:40.649 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_list on pid 133654' 2023-07-12T15:29:40.649 INFO:tasks.workunit.client.0.smithi069.stdout:test api_list on pid 133654 2023-07-12T15:29:40.650 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133654 2023-07-12T15:29:40.650 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.650 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.650 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-07-12T15:29:40.650 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_list 2023-07-12T15:29:40.650 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_list: /' 2023-07-12T15:29:40.651 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_list.log 2023-07-12T15:29:40.651 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_lock 2023-07-12T15:29:40.651 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_lock' 2023-07-12T15:29:40.652 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.652 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_lock 2023-07-12T15:29:40.654 INFO:tasks.workunit.client.0.smithi069.stdout:test api_lock on pid 133662 2023-07-12T15:29:40.654 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_lock 2023-07-12T15:29:40.654 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133662 2023-07-12T15:29:40.655 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_lock on pid 133662' 2023-07-12T15:29:40.655 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133662 2023-07-12T15:29:40.655 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.655 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.657 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-07-12T15:29:40.659 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_lock.log 2023-07-12T15:29:40.659 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_lock_pp 2023-07-12T15:29:40.659 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_lock_pp' 2023-07-12T15:29:40.659 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_lock: /' 2023-07-12T15:29:40.660 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_lock 2023-07-12T15:29:40.667 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_lock_pp 2023-07-12T15:29:40.667 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.673 INFO:tasks.workunit.client.0.smithi069.stdout:test api_lock_pp on pid 133701 2023-07-12T15:29:40.673 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_lock_pp 2023-07-12T15:29:40.674 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133701 2023-07-12T15:29:40.674 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_lock_pp on pid 133701' 2023-07-12T15:29:40.674 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133701 2023-07-12T15:29:40.674 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.674 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.674 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-07-12T15:29:40.674 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_misc 2023-07-12T15:29:40.675 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_misc' 2023-07-12T15:29:40.682 INFO:tasks.workunit.client.0.smithi069.stdout:test api_misc on pid 133739 2023-07-12T15:29:40.683 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_lock_pp 2023-07-12T15:29:40.683 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-07-12T15:29:40.683 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_lock_pp: /' 2023-07-12T15:29:40.683 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_misc 2023-07-12T15:29:40.683 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.683 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_misc 2023-07-12T15:29:40.684 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133739 2023-07-12T15:29:40.684 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_misc on pid 133739' 2023-07-12T15:29:40.684 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133739 2023-07-12T15:29:40.684 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.684 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_misc_pp 2023-07-12T15:29:40.685 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_misc_pp' 2023-07-12T15:29:40.685 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-07-12T15:29:40.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_misc_pp 2023-07-12T15:29:40.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.686 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_misc 2023-07-12T15:29:40.687 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_misc_pp 2023-07-12T15:29:40.687 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_misc.log 2023-07-12T15:29:40.687 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133754 2023-07-12T15:29:40.687 INFO:tasks.workunit.client.0.smithi069.stdout:test api_misc_pp on pid 133754 2023-07-12T15:29:40.688 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_misc_pp on pid 133754' 2023-07-12T15:29:40.688 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133754 2023-07-12T15:29:40.688 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.689 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.690 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-07-12T15:29:40.691 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_misc: /' 2023-07-12T15:29:40.691 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_misc_pp 2023-07-12T15:29:40.691 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-07-12T15:29:40.693 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_tier_pp 2023-07-12T15:29:40.693 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_tier_pp' 2023-07-12T15:29:40.695 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_misc_pp: /' 2023-07-12T15:29:40.695 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_tier_pp 2023-07-12T15:29:40.696 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.698 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_tier_pp 2023-07-12T15:29:40.700 INFO:tasks.workunit.client.0.smithi069.stdout:test api_tier_pp on pid 133786 2023-07-12T15:29:40.700 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133786 2023-07-12T15:29:40.701 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_tier_pp on pid 133786' 2023-07-12T15:29:40.701 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133786 2023-07-12T15:29:40.701 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.701 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.701 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-07-12T15:29:40.701 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_pool 2023-07-12T15:29:40.701 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_pool' 2023-07-12T15:29:40.702 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_tier_pp 2023-07-12T15:29:40.703 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-07-12T15:29:40.710 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_tier_pp: /' 2023-07-12T15:29:40.719 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.719 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_pool 2023-07-12T15:29:40.720 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_pool 2023-07-12T15:29:40.720 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133815 2023-07-12T15:29:40.720 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_pool on pid 133815' 2023-07-12T15:29:40.721 INFO:tasks.workunit.client.0.smithi069.stdout:test api_pool on pid 133815 2023-07-12T15:29:40.721 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133815 2023-07-12T15:29:40.721 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.722 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.722 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_snapshots 2023-07-12T15:29:40.722 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_snapshots' 2023-07-12T15:29:40.722 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-07-12T15:29:40.722 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.723 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_pool 2023-07-12T15:29:40.723 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_pool.log 2023-07-12T15:29:40.724 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_snapshots 2023-07-12T15:29:40.724 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_snapshots 2023-07-12T15:29:40.724 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133832 2023-07-12T15:29:40.725 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_snapshots on pid 133832' 2023-07-12T15:29:40.725 INFO:tasks.workunit.client.0.smithi069.stdout:test api_snapshots on pid 133832 2023-07-12T15:29:40.726 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133832 2023-07-12T15:29:40.726 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.726 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.726 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-07-12T15:29:40.726 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_snapshots_pp 2023-07-12T15:29:40.726 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_snapshots_pp' 2023-07-12T15:29:40.727 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_snapshots_pp 2023-07-12T15:29:40.727 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.727 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-07-12T15:29:40.728 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_snapshots_pp 2023-07-12T15:29:40.728 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133844 2023-07-12T15:29:40.728 INFO:tasks.workunit.client.0.smithi069.stdout:test api_snapshots_pp on pid 133844 2023-07-12T15:29:40.729 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_snapshots_pp on pid 133844' 2023-07-12T15:29:40.729 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133844 2023-07-12T15:29:40.729 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.729 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-07-12T15:29:40.729 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_stat 2023-07-12T15:29:40.730 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_stat' 2023-07-12T15:29:40.730 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_snapshots 2023-07-12T15:29:40.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.731 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_pool: /' 2023-07-12T15:29:40.731 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_snapshots: /' 2023-07-12T15:29:40.731 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_snapshots_pp 2023-07-12T15:29:40.731 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-07-12T15:29:40.731 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_stat 2023-07-12T15:29:40.732 INFO:tasks.workunit.client.0.smithi069.stdout:test api_stat on pid 133857 2023-07-12T15:29:40.733 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [==========] Running 11 tests from 4 test suites. 2023-07-12T15:29:40.733 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] Global test environment set-up. 2023-07-12T15:29:40.733 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-07-12T15:29:40.733 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-07-12T15:29:40.733 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-07-12T15:29:40.733 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-07-12T15:29:40.733 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.734 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscConnectFailure 2023-07-12T15:29:40.734 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-07-12T15:29:40.734 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 0 2023-07-12T15:29:40.734 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.726+0000 7f4bb14208c0 10 monclient: get_monmap_and_config 2023-07-12T15:29:40.734 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.726+0000 7f4bb14208c0 10 monclient: build_initial_monmap 2023-07-12T15:29:40.734 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.726+0000 7f4bb14208c0 10 monclient: monmap: 2023-07-12T15:29:40.734 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: epoch 0 2023-07-12T15:29:40.734 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: fsid dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:29:40.735 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: last_changed 2023-07-12T15:29:40.726857+0000 2023-07-12T15:29:40.735 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: created 2023-07-12T15:29:40.726857+0000 2023-07-12T15:29:40.735 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: min_mon_release 0 (unknown) 2023-07-12T15:29:40.735 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: election_strategy: 1 2023-07-12T15:29:40.735 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-07-12T15:29:40.735 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.noname-b 2023-07-12T15:29:40.735 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-07-12T15:29:40.736 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.736 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 1 Processor -- start 2023-07-12T15:29:40.736 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 1 -- start start 2023-07-12T15:29:40.736 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 10 monclient: init 2023-07-12T15:29:40.736 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 10 monclient: _reopen_session rank -1 2023-07-12T15:29:40.736 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 10 monclient: _add_conns ranks=[0,1,2] 2023-07-12T15:29:40.736 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a289769af0 0x55a289769ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:29:40.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 10 monclient(hunting): picked mon.noname-a con 0x55a289769af0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-07-12T15:29:40.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a28976a400 0x55a289800170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:29:40.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 10 monclient(hunting): picked mon.noname-b con 0x55a28976a400 addr [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] 2023-07-12T15:29:40.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a2898006b0 0x55a289802a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:29:40.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 10 monclient(hunting): picked mon.noname-c con 0x55a2898006b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-07-12T15:29:40.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 10 monclient(hunting): start opening mon connection 2023-07-12T15:29:40.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a288bbc2c0 con 0x55a289769af0 2023-07-12T15:29:40.737 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 10 monclient(hunting): start opening mon connection 2023-07-12T15:29:40.738 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a288b58030 con 0x55a28976a400 2023-07-12T15:29:40.738 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_stat 2023-07-12T15:29:40.738 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133857 2023-07-12T15:29:40.738 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_stat on pid 133857' 2023-07-12T15:29:40.738 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133857 2023-07-12T15:29:40.739 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.739 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 10 monclient(hunting): start opening mon connection 2023-07-12T15:29:40.739 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a288b57cd0 con 0x55a2898006b0 2023-07-12T15:29:40.740 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 10 monclient(hunting): _renew_subs 2023-07-12T15:29:40.740 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bb14208c0 10 monclient(hunting): authenticate will time out at 1473.333527s 2023-07-12T15:29:40.740 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bae726700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a289769af0 0x55a289769ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:29:40.740 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bae726700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a289769af0 0x55a289769ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41678/0 (socket says 172.21.15.69:41678) 2023-07-12T15:29:40.740 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bae726700 1 -- 172.21.15.69:0/1853169775 learned_addr learned my addr 172.21.15.69:0/1853169775 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:29:40.740 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bae726700 10 monclient(hunting): get_auth_request con 0x55a289769af0 auth_method 0 2023-07-12T15:29:40.740 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bae726700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-07-12T15:29:40.741 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bae726700 10 monclient(hunting): _init_auth method 2 2023-07-12T15:29:40.741 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4bae726700 10 monclient(hunting): _init_auth creating new auth 2023-07-12T15:29:40.741 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4badf25700 1 --2- 172.21.15.69:0/1853169775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a28976a400 0x55a289800170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:29:40.741 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4badf25700 10 monclient(hunting): get_auth_request con 0x55a28976a400 auth_method 0 2023-07-12T15:29:40.741 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4badf25700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-07-12T15:29:40.741 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4badf25700 10 monclient(hunting): _init_auth method 2 2023-07-12T15:29:40.741 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.731+0000 7f4badf25700 10 monclient(hunting): _init_auth creating new auth 2023-07-12T15:29:40.742 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bae726700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-07-12T15:29:40.742 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bae726700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-07-12T15:29:40.742 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bae726700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-07-12T15:29:40.742 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-07-12T15:29:40.742 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-07-12T15:29:40.742 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-07-12T15:29:40.742 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 10 monclient(hunting): handle_auth_done global_id 24700 payload 274 2023-07-12T15:29:40.742 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 1 -- 172.21.15.69:0/1853169775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a2898006b0 msgr2=0x55a289802a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-12T15:29:40.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 1 --2- 172.21.15.69:0/1853169775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a2898006b0 0x55a289802a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:29:40.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 1 -- 172.21.15.69:0/1853169775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a289769af0 msgr2=0x55a289769ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-12T15:29:40.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 1 --2- 172.21.15.69:0/1853169775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a289769af0 0x55a289769ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:29:40.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 10 monclient: _finish_hunting 0 2023-07-12T15:29:40.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 1 monclient: found mon.noname-b 2023-07-12T15:29:40.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-07-12T15:29:40.743 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.145:3300/0 2023-07-12T15:29:40.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 1 -- 172.21.15.69:0/1853169775 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a288b5a4a0 con 0x55a28976a400 2023-07-12T15:29:40.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4badf25700 1 --2- 172.21.15.69:0/1853169775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a28976a400 0x55a289800170 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000a4a0 tx=0x7f4ba000a7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1cd70df862a119 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:29:40.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bad724700 1 -- 172.21.15.69:0/1853169775 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba000f790 con 0x55a28976a400 2023-07-12T15:29:40.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bad724700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-07-12T15:29:40.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bad724700 10 monclient: got monmap 3 from mon.noname-b (according to old e3) 2023-07-12T15:29:40.744 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bad724700 10 monclient: dump: 2023-07-12T15:29:40.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: epoch 3 2023-07-12T15:29:40.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: fsid dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:29:40.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: last_changed 2023-07-12T15:24:21.092131+0000 2023-07-12T15:29:40.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: created 2023-07-12T15:22:46.541643+0000 2023-07-12T15:29:40.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: min_mon_release 18 (reef) 2023-07-12T15:29:40.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: election_strategy: 1 2023-07-12T15:29:40.745 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-07-12T15:29:40.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2023-07-12T15:29:40.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-07-12T15:29:40.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bad724700 10 monclient: _finish_auth 0 2023-07-12T15:29:40.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bad724700 10 monclient: _check_auth_tickets 2023-07-12T15:29:40.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bad724700 20 monclient: _check_auth_rotating not needed by client.admin 2023-07-12T15:29:40.746 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bad724700 1 -- 172.21.15.69:0/1853169775 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f4ba000fd80 con 0x55a28976a400 2023-07-12T15:29:40.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 5 monclient: authenticate success, global_id 24700 2023-07-12T15:29:40.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 20 monclient: get_monmap_and_config waiting for monmap|config 2023-07-12T15:29:40.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bad724700 10 monclient: handle_config config(25 keys) v1 2023-07-12T15:29:40.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bad724700 1 -- 172.21.15.69:0/1853169775 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0010d40 con 0x55a28976a400 2023-07-12T15:29:40.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bad724700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-07-12T15:29:40.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bad724700 10 monclient: got monmap 3 from mon.b (according to old e3) 2023-07-12T15:29:40.747 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bad724700 10 monclient: dump: 2023-07-12T15:29:40.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: epoch 3 2023-07-12T15:29:40.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: fsid dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:29:40.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: last_changed 2023-07-12T15:24:21.092131+0000 2023-07-12T15:29:40.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: created 2023-07-12T15:22:46.541643+0000 2023-07-12T15:29:40.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: min_mon_release 18 (reef) 2023-07-12T15:29:40.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: election_strategy: 1 2023-07-12T15:29:40.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-07-12T15:29:40.748 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2023-07-12T15:29:40.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-07-12T15:29:40.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 monclient: get_monmap_and_config success 2023-07-12T15:29:40.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 4 set_mon_vals no callback set 2023-07-12T15:29:40.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals auth_debug = true 2023-07-12T15:29:40.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals container_image = quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c4be3de0cb572873b18bf4d7ff19efd4676dcd86f150f2b38e50b5e2946cf6ef 2023-07-12T15:29:40.749 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals debug_asserts_on_shutdown = true 2023-07-12T15:29:40.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 20 set_mon_vals debug_ms = 1/1 (no change) 2023-07-12T15:29:40.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals log_to_file = true 2023-07-12T15:29:40.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals log_to_journald = false 2023-07-12T15:29:40.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 20 set_mon_vals log_to_stderr = false (no change) 2023-07-12T15:29:40.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals mon_allow_pool_delete = true 2023-07-12T15:29:40.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals mon_clock_drift_allowed = 1.000000 2023-07-12T15:29:40.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals mon_cluster_log_file_level = debug 2023-07-12T15:29:40.750 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals mon_election_default_strategy = 3 2023-07-12T15:29:40.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals mon_max_pg_per_osd = 10000 2023-07-12T15:29:40.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals mon_pg_warn_max_object_skew = 0.000000 2023-07-12T15:29:40.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals mon_warn_on_crush_straw_calc_version_zero = false 2023-07-12T15:29:40.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals mon_warn_on_legacy_crush_tunables = false 2023-07-12T15:29:40.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals mon_warn_on_osd_down_out_interval_zero = false 2023-07-12T15:29:40.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals mon_warn_on_pool_pg_num_not_power_of_two = false 2023-07-12T15:29:40.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals mon_warn_on_too_few_osds = false 2023-07-12T15:29:40.751 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals ms_bind_msgr1 = false 2023-07-12T15:29:40.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals ms_bind_msgr2 = true 2023-07-12T15:29:40.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals ms_die_on_bug = true 2023-07-12T15:29:40.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals ms_die_on_old_message = true 2023-07-12T15:29:40.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals ms_type = async 2023-07-12T15:29:40.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals osd_pool_default_erasure_code_profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-07-12T15:29:40.752 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:40 smithi069 ceph-mon[94241]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:40.753 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:40 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:40.753 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:40 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:40.753 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:40 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:40.753 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:40 smithi069 ceph-mon[94241]: Deploying daemon node-exporter.b on smithi145 2023-07-12T15:29:40.753 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:40 smithi069 ceph-mon[102777]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:40.754 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:40 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:40.754 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:40 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:40.754 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:40 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:40.754 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:40 smithi069 ceph-mon[102777]: Deploying daemon node-exporter.b on smithi145 2023-07-12T15:29:40.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.732+0000 7f4bb14208c0 10 set_mon_vals osd_pool_default_pg_autoscale_mode = off 2023-07-12T15:29:40.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.733+0000 7f4bb14208c0 10 monclient: shutdown 2023-07-12T15:29:40.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.733+0000 7f4bb14208c0 20 monclient: shutdown discarding 0 pending message(s) 2023-07-12T15:29:40.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.733+0000 7f4bb14208c0 1 -- 172.21.15.69:0/1853169775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a28976a400 msgr2=0x55a289800170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:29:40.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.733+0000 7f4bb14208c0 1 --2- 172.21.15.69:0/1853169775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a28976a400 0x55a289800170 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000a4a0 tx=0x7f4ba000a7b0 comp rx=0 tx=0).stop 2023-07-12T15:29:40.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.733+0000 7f4bb14208c0 1 -- 172.21.15.69:0/1853169775 shutdown_connections 2023-07-12T15:29:40.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.733+0000 7f4bb14208c0 1 --2- 172.21.15.69:0/1853169775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a2898006b0 0x55a289802a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:29:40.755 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.733+0000 7f4bb14208c0 1 --2- 172.21.15.69:0/1853169775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a289769af0 0x55a289769ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:29:40.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.733+0000 7f4bb14208c0 1 --2- 172.21.15.69:0/1853169775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a28976a400 0x55a289800170 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:29:40.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.733+0000 7f4bb14208c0 1 -- 172.21.15.69:0/1853169775 >> 172.21.15.69:0/1853169775 conn(0x55a289765570 msgr2=0x55a289767980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:29:40.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.733+0000 7f4bb14208c0 1 -- 172.21.15.69:0/1853169775 shutdown_connections 2023-07-12T15:29:40.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.733+0000 7f4bb14208c0 1 -- 172.21.15.69:0/1853169775 wait complete. 2023-07-12T15:29:40.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.734+0000 7f4bb14208c0 10 monclient: build_initial_monmap 2023-07-12T15:29:40.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.734+0000 7f4bb14208c0 10 monclient: monmap: 2023-07-12T15:29:40.756 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: epoch 0 2023-07-12T15:29:40.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: fsid 00000000-0000-0000-0000-000000000000 2023-07-12T15:29:40.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: last_changed 0.000000 2023-07-12T15:29:40.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: created 0.000000 2023-07-12T15:29:40.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: min_mon_release 0 (unknown) 2023-07-12T15:29:40.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: election_strategy: 1 2023-07-12T15:29:40.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2023-07-12T15:29:40.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.noname-b 2023-07-12T15:29:40.757 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2023-07-12T15:29:40.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.734+0000 7f4bb14208c0 1 Processor -- start 2023-07-12T15:29:40.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.734+0000 7f4bb14208c0 1 -- start start 2023-07-12T15:29:40.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.734+0000 7f4bb14208c0 10 monclient: init 2023-07-12T15:29:40.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 10 monclient: _reopen_session rank -1 2023-07-12T15:29:40.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 10 monclient: _add_conns ranks=[0,1,2] 2023-07-12T15:29:40.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a289769af0 0x55a2897fa250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:29:40.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 10 monclient(hunting): picked mon.noname-a con 0x55a289769af0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2023-07-12T15:29:40.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a28976a400 0x55a2897fa790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:29:40.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 10 monclient(hunting): picked mon.noname-b con 0x55a28976a400 addr [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] 2023-07-12T15:29:40.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a2898006b0 0x55a2897facd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:29:40.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 10 monclient(hunting): picked mon.noname-c con 0x55a2898006b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2023-07-12T15:29:40.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 10 monclient(hunting): start opening mon connection 2023-07-12T15:29:40.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a288b4efe0 con 0x55a289769af0 2023-07-12T15:29:40.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 10 monclient(hunting): start opening mon connection 2023-07-12T15:29:40.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a288ce6e30 con 0x55a28976a400 2023-07-12T15:29:40.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 10 monclient(hunting): start opening mon connection 2023-07-12T15:29:40.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a288ce6ff0 con 0x55a2898006b0 2023-07-12T15:29:40.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 10 monclient(hunting): _renew_subs 2023-07-12T15:29:40.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bb14208c0 10 monclient(hunting): authenticate will time out at 1473.337233s 2023-07-12T15:29:40.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a289769af0 0x55a2897fa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:29:40.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a289769af0 0x55a2897fa250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41684/0 (socket says 172.21.15.69:41684) 2023-07-12T15:29:40.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 1 -- 172.21.15.69:0/2979290620 learned_addr learned my addr 172.21.15.69:0/2979290620 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:29:40.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 10 monclient(hunting): get_auth_request con 0x55a289769af0 auth_method 0 2023-07-12T15:29:40.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-07-12T15:29:40.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 10 monclient(hunting): _init_auth method 2 2023-07-12T15:29:40.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 10 monclient(hunting): _init_auth creating new auth 2023-07-12T15:29:40.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-07-12T15:29:40.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-07-12T15:29:40.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-07-12T15:29:40.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4baef27700 1 --2- 172.21.15.69:0/2979290620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a2898006b0 0x55a2897facd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:29:40.762 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-07-12T15:29:40.763 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-07-12T15:29:40.763 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_stat_pp 2023-07-12T15:29:40.763 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_stat_pp' 2023-07-12T15:29:40.763 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_stat 2023-07-12T15:29:40.763 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_stat_pp 2023-07-12T15:29:40.763 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_stat.log 2023-07-12T15:29:40.763 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_stat: /' 2023-07-12T15:29:40.764 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.764 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_stat_pp 2023-07-12T15:29:40.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4baef27700 10 monclient(hunting): get_auth_requetest api_stat_pp on pid 133901 2023-07-12T15:29:40.765 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133901 2023-07-12T15:29:40.765 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_stat_pp on pid 133901' 2023-07-12T15:29:40.765 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133901 2023-07-12T15:29:40.765 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.765 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.765 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_watch_notify 2023-07-12T15:29:40.765 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-07-12T15:29:40.766 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_watch_notify' 2023-07-12T15:29:40.766 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_watch_notify 2023-07-12T15:29:40.766 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.766 INFO:tasks.workunit.client.0.smithi069.stdout:test api_watch_notify on pid 133927 2023-07-12T15:29:40.767 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_watch_notify 2023-07-12T15:29:40.767 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-07-12T15:29:40.767 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_stat_pp: /' 2023-07-12T15:29:40.767 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133927 2023-07-12T15:29:40.768 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_watch_notify on pid 133927' 2023-07-12T15:29:40.768 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133927 2023-07-12T15:29:40.768 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.768 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.768 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-07-12T15:29:40.768 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_stat_pp 2023-07-12T15:29:40.768 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-07-12T15:29:40.769 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_watch_notify_pp 2023-07-12T15:29:40.769 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_watch_notify_pp' 2023-07-12T15:29:40.769 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_watch_notify 2023-07-12T15:29:40.769 INFO:tasks.workunit.client.0.smithi069.stdout:test api_watch_notify_pp on pid 133939 2023-07-12T15:29:40.770 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_watch_notify: /' 2023-07-12T15:29:40.770 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_watch_notify_pp 2023-07-12T15:29:40.770 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.770 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_watch_notify_pp 2023-07-12T15:29:40.770 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133939 2023-07-12T15:29:40.771 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_watch_notify_pp on pid 133939' 2023-07-12T15:29:40.771 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133939 2023-07-12T15:29:40.771 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.771 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cmd 2023-07-12T15:29:40.771 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cmd' 2023-07-12T15:29:40.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cmd 2023-07-12T15:29:40.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.772 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cmd 2023-07-12T15:29:40.772 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133951 2023-07-12T15:29:40.772 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cmd on pid 133951' 2023-07-12T15:29:40.773 INFO:tasks.workunit.client.0.smithi069.stdout:test api_cmd on pid 133951 2023-07-12T15:29:40.773 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133951 2023-07-12T15:29:40.773 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.774 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.774 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-07-12T15:29:40.774 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-07-12T15:29:40.774 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cmd_pp 2023-07-12T15:29:40.774 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cmd_pp' 2023-07-12T15:29:40.774 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cmd.log 2023-07-12T15:29:40.774 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cmd: /' 2023-07-12T15:29:40.775 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_watch_notify_pp 2023-07-12T15:29:40.775 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cmd_pp 2023-07-12T15:29:40.775 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-07-12T15:29:40.775 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.776 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-07-12T15:29:40.776 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cmd_pp 2023-07-12T15:29:40.776 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133969 2023-07-12T15:29:40.777 INFO:tasks.workunit.client.0.smithi069.stdout:test api_cmd_pp on pid 133969 2023-07-12T15:29:40.777 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cmd_pp on pid 133969' 2023-07-12T15:29:40.777 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133969 2023-07-12T15:29:40.778 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.778 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.778 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-07-12T15:29:40.786 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-07-12T15:29:40.786 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cmd 2023-07-12T15:29:40.786 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_service 2023-07-12T15:29:40.786 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_service' 2023-07-12T15:29:40.786 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-07-12T15:29:40.787 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_service 2023-07-12T15:29:40.787 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.787 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_service 2023-07-12T15:29:40.787 INFO:tasks.workunit.client.0.smithi069.stdout:test api_service on pid 133991 2023-07-12T15:29:40.788 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133991 2023-07-12T15:29:40.788 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_service on pid 133991' 2023-07-12T15:29:40.788 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=133991 2023-07-12T15:29:40.788 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.788 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.788 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-07-12T15:29:40.789 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cmd_pp 2023-07-12T15:29:40.789 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_service 2023-07-12T15:29:40.789 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_service_pp 2023-07-12T15:29:40.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_service_pp' 2023-07-12T15:29:40.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_service.log 2023-07-12T15:29:40.803 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_service: /' 2023-07-12T15:29:40.820 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_service_pp 2023-07-12T15:29:40.820 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.822 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_service_pp 2023-07-12T15:29:40.822 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134051 2023-07-12T15:29:40.822 INFO:tasks.workunit.client.0.smithi069.stdout:test api_service_pp on pid 134051 2023-07-12T15:29:40.823 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_service_pp on pid 134051' 2023-07-12T15:29:40.823 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=134051 2023-07-12T15:29:40.823 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.823 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.824 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-07-12T15:29:40.824 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_c_write_operations 2023-07-12T15:29:40.824 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_c_write_operations' 2023-07-12T15:29:40.824 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_c_write_operations 2023-07-12T15:29:40.825 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-07-12T15:29:40.825 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_service_pp 2023-07-12T15:29:40.825 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.825 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_service_pp: /' 2023-07-12T15:29:40.825 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_c_write_operations 2023-07-12T15:29:40.826 INFO:tasks.workunit.client.0.smithi069.stdout:test api_c_write_operations on pid 134067 2023-07-12T15:29:40.826 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134067 2023-07-12T15:29:40.826 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_c_write_operations on pid 134067' 2023-07-12T15:29:40.826 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=134067 2023-07-12T15:29:40.827 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.827 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.827 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-07-12T15:29:40.827 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_c_read_operations 2023-07-12T15:29:40.828 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_c_read_operations' 2023-07-12T15:29:40.828 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_c_write_operations 2023-07-12T15:29:40.828 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-07-12T15:29:40.828 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.830 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-07-12T15:29:40.830 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_c_read_operations 2023-07-12T15:29:40.830 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_c_read_operations 2023-07-12T15:29:40.831 INFO:tasks.workunit.client.0.smithi069.stdout:test api_c_read_operations on pid 134075 2023-07-12T15:29:40.831 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134075 2023-07-12T15:29:40.831 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_c_read_operations on pid 134075' 2023-07-12T15:29:40.831 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=134075 2023-07-12T15:29:40.831 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.832 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.832 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-07-12T15:29:40.832 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cls_remote_reads 2023-07-12T15:29:40.833 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cls_remote_reads' 2023-07-12T15:29:40.833 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cls_remote_reads 2023-07-12T15:29:40.833 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.834 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_c_read_operations 2023-07-12T15:29:40.835 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-07-12T15:29:40.835 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-07-12T15:29:40.836 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cls_remote_reads 2023-07-12T15:29:40.836 INFO:tasks.workunit.client.0.smithi069.stdout:test api_cls_remote_reads on pid 134090 2023-07-12T15:29:40.836 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134090 2023-07-12T15:29:40.837 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cls_remote_reads on pid 134090' 2023-07-12T15:29:40.837 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=134090 2023-07-12T15:29:40.837 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.837 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.837 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-07-12T15:29:40.838 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cls_remote_reads 2023-07-12T15:29:40.838 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-07-12T15:29:40.839 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s list_parallel 2023-07-12T15:29:40.839 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' list_parallel' 2023-07-12T15:29:40.839 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.840 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo list_parallel 2023-07-12T15:29:40.840 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-07-12T15:29:40.841 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=list_parallel 2023-07-12T15:29:40.841 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134098 2023-07-12T15:29:40.841 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test list_parallel on pid 134098' 2023-07-12T15:29:40.842 INFO:tasks.workunit.client.0.smithi069.stdout:test list_parallel on pid 134098 2023-07-12T15:29:40.842 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=134098 2023-07-12T15:29:40.842 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.843 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.843 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-07-12T15:29:40.843 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s open_pools_parallel 2023-07-12T15:29:40.843 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' open_pools_parallel' 2023-07-12T15:29:40.844 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ list_parallel: /' 2023-07-12T15:29:40.844 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo open_pools_parallel 2023-07-12T15:29:40.844 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.845 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_list_parallel 2023-07-12T15:29:40.847 INFO:tasks.workunit.client.0.smithi069.stdout:test open_pools_parallel on pid 134107 2023-07-12T15:29:40.848 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=open_pools_parallel 2023-07-12T15:29:40.848 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134107 2023-07-12T15:29:40.849 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test open_pools_parallel on pid 134107' 2023-07-12T15:29:40.849 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=134107 2023-07-12T15:29:40.849 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-07-12T15:29:40.850 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.850 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_list_parallel.log 2023-07-12T15:29:40.851 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:40 smithi145 bash[123052]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2023-07-12T15:29:40.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:40 smithi145 ceph-mon[95801]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:29:40.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:40 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:40.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:40 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:40.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:40 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:40.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:40 smithi145 ceph-mon[95801]: Deploying daemon node-exporter.b on smithi145 2023-07-12T15:29:40.853 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s delete_pools_parallel 2023-07-12T15:29:40.853 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' delete_pools_parallel' 2023-07-12T15:29:40.853 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-07-12T15:29:40.854 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2023-07-12T15:29:40.854 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo delete_pools_parallel 2023-07-12T15:29:40.854 INFO:tasks.workunit.client.0.smithi069.stdout:test delete_pools_parallel on pid 134117 2023-07-12T15:29:40.855 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=delete_pools_parallel 2023-07-12T15:29:40.855 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134117 2023-07-12T15:29:40.855 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test delete_pools_parallel on pid 134117' 2023-07-12T15:29:40.856 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=134117 2023-07-12T15:29:40.856 INFO:tasks.workunit.client.0.smithi069.stderr:+ ret=0 2023-07-12T15:29:40.856 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2023-07-12T15:29:40.856 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:29:40.856 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133832 2023-07-12T15:29:40.856 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133832 2023-07-12T15:29:40.857 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-07-12T15:29:40.857 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-07-12T15:29:40.860 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-07-12T15:29:40.862 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-07-12T15:29:40.863 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-07-12T15:29:40.864 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_open_pools_parallel 2023-07-12T15:29:40.870 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_delete_pools_parallel 2023-07-12T15:29:40.981 INFO:tasks.workunit.client.0.smithi069.stdout:st con 0x55a2898006b0 auth_method 0 2023-07-12T15:29:40.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4baef27700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-07-12T15:29:40.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4baef27700 10 monclient(hunting): _init_auth method 2 2023-07-12T15:29:40.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4baef27700 10 monclient(hunting): _init_auth creating new auth 2023-07-12T15:29:40.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 10 monclient(hunting): handle_auth_done global_id 14823 payload 824 2023-07-12T15:29:40.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 1 -- 172.21.15.69:0/2979290620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a2898006b0 msgr2=0x55a2897facd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:29:40.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 1 --2- 172.21.15.69:0/2979290620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a2898006b0 0x55a2897facd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:29:40.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 1 -- 172.21.15.69:0/2979290620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a28976a400 msgr2=0x55a2897fa790 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-12T15:29:40.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 1 --2- 172.21.15.69:0/2979290620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a28976a400 0x55a2897fa790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:29:40.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 10 monclient: _finish_hunting 0 2023-07-12T15:29:40.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 1 monclient: found mon.noname-a 2023-07-12T15:29:40.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-07-12T15:29:40.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2023-07-12T15:29:40.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.735+0000 7f4bae726700 1 -- 172.21.15.69:0/2979290620 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba000a100 con 0x55a289769af0 2023-07-12T15:29:40.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.736+0000 7f4bae726700 1 --2- 172.21.15.69:0/2979290620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a289769af0 0x55a2897fa250 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800a790 tx=0x7f4b9800a870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:29:40.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4b9effd700 1 -- 172.21.15.69:0/2979290620 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b98010070 con 0x55a289769af0 2023-07-12T15:29:40.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4b9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-07-12T15:29:40.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4b9effd700 10 monclient: got monmap 3 from mon.noname-a (according to old e3) 2023-07-12T15:29:40.984 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4b9effd700 10 monclient: dump: 2023-07-12T15:29:40.984 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: epoch 3 2023-07-12T15:29:40.984 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: fsid dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:29:40.984 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: last_changed 2023-07-12T15:24:21.092131+0000 2023-07-12T15:29:40.984 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: created 2023-07-12T15:22:46.541643+0000 2023-07-12T15:29:40.984 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: min_mon_release 18 (reef) 2023-07-12T15:29:40.984 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: election_strategy: 1 2023-07-12T15:29:40.985 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-07-12T15:29:40.985 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2023-07-12T15:29:40.985 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2023-07-12T15:29:40.985 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.985 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4b9effd700 10 monclient: _finish_auth 0 2023-07-12T15:29:40.985 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4b9effd700 10 monclient: _check_auth_tickets 2023-07-12T15:29:41.850 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:41 smithi145 bash[123052]: Getting image source signatures 2023-07-12T15:29:41.851 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:41 smithi145 bash[123052]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2023-07-12T15:29:41.851 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:41 smithi145 bash[123052]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2023-07-12T15:29:41.851 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:41 smithi145 bash[123052]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2023-07-12T15:29:41.851 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:41 smithi145 bash[123052]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2023-07-12T15:29:41.851 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:41 smithi145 bash[123052]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2023-07-12T15:29:41.851 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:41 smithi145 bash[123052]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2023-07-12T15:29:42.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:42.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:42.798 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:43.100 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:42 smithi145 bash[123052]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2023-07-12T15:29:43.101 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:42 smithi145 bash[123052]: Writing manifest to image destination 2023-07-12T15:29:43.101 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:42 smithi145 bash[123052]: Storing signatures 2023-07-12T15:29:43.583 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 podman[123052]: 2023-07-12T15:29:43.583 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.559Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2023-07-12T15:29:43.583 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.559Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2023-07-12T15:29:43.583 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.559Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2023-07-12T15:29:43.583 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.559Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2023-07-12T15:29:43.584 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.559Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2023-07-12T15:29:43.584 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.559Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2023-07-12T15:29:43.584 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2023-07-12T15:29:43.584 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=arp 2023-07-12T15:29:43.584 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=bcache 2023-07-12T15:29:43.584 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=bonding 2023-07-12T15:29:43.584 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=btrfs 2023-07-12T15:29:43.585 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=conntrack 2023-07-12T15:29:43.585 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=cpu 2023-07-12T15:29:43.585 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=cpufreq 2023-07-12T15:29:43.585 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=diskstats 2023-07-12T15:29:43.585 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=dmi 2023-07-12T15:29:43.585 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=edac 2023-07-12T15:29:43.585 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=entropy 2023-07-12T15:29:43.586 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=fibrechannel 2023-07-12T15:29:43.586 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=filefd 2023-07-12T15:29:43.586 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=filesystem 2023-07-12T15:29:43.586 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=hwmon 2023-07-12T15:29:43.586 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=infiniband 2023-07-12T15:29:43.586 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=ipvs 2023-07-12T15:29:43.587 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=loadavg 2023-07-12T15:29:43.587 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=mdadm 2023-07-12T15:29:43.587 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=meminfo 2023-07-12T15:29:43.587 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=netclass 2023-07-12T15:29:43.588 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=netdev 2023-07-12T15:29:43.588 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=netstat 2023-07-12T15:29:43.588 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=nfs 2023-07-12T15:29:43.588 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=nfsd 2023-07-12T15:29:43.588 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=nvme 2023-07-12T15:29:43.589 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=os 2023-07-12T15:29:43.589 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2023-07-12T15:29:43.589 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=pressure 2023-07-12T15:29:43.589 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=rapl 2023-07-12T15:29:43.589 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=schedstat 2023-07-12T15:29:43.589 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=selinux 2023-07-12T15:29:43.590 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=sockstat 2023-07-12T15:29:43.590 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=softnet 2023-07-12T15:29:43.590 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=stat 2023-07-12T15:29:43.590 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=tapestats 2023-07-12T15:29:43.590 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=textfile 2023-07-12T15:29:43.590 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=thermal_zone 2023-07-12T15:29:43.590 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=time 2023-07-12T15:29:43.591 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=udp_queues 2023-07-12T15:29:43.591 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=uname 2023-07-12T15:29:43.591 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=vmstat 2023-07-12T15:29:43.591 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=xfs 2023-07-12T15:29:43.591 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=node_exporter.go:117 level=info collector=zfs 2023-07-12T15:29:43.591 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2023-07-12T15:29:43.591 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b[123103]: ts=2023-07-12T15:29:43.560Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2023-07-12T15:29:43.592 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 bash[123052]: 04cf3658ed6aeb159caf67453946cf40d2a0d3289edc4d6863924c8d77b56616 2023-07-12T15:29:44.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:29:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:29:44.793 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:29:43 smithi145 systemd[1]: Started Ceph node-exporter.b for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:29:45.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/201619042' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-133627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-133656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1921160841' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-133635-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2784286066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-133619-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:45.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:44 smithi145 ceph-mon[95801]: osdmap e72: 8 total, 8 up, 8 in 2023-07-12T15:29:45.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/201619042' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-133627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-133656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1921160841' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-133635-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2784286066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-133619-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:44 smithi069 ceph-mon[102777]: osdmap e72: 8 total, 8 up, 8 in 2023-07-12T15:29:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/201619042' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-133627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-133656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1921160841' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-133635-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2784286066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-133619-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:44 smithi069 ceph-mon[94241]: osdmap e72: 8 total, 8 up, 8 in 2023-07-12T15:29:46.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:29:46] "GET /metrics HTTP/1.1" 200 240653 "" "Prometheus/2.43.0" 2023-07-12T15:29:46.632 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12 api_asio: [==========] Running 12 tests from 1 test suite. 2023-07-12T15:29:46.632 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] Global test environment set-up. 2023-07-12T15:29:46.632 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] 12 tests from AsioRados 2023-07-12T15:29:46.633 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-07-12T15:29:46.633 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (11 ms) 2023-07-12T15:29:46.633 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-07-12T15:29:46.633 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (2 ms) 2023-07-12T15:29:46.633 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-07-12T15:29:46.633 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2023-07-12T15:29:46.633 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-07-12T15:29:46.633 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (4 ms) 2023-07-12T15:29:46.634 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-07-12T15:29:46.634 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (1 ms) 2023-07-12T15:29:46.634 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-07-12T15:29:46.634 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (2 ms) 2023-07-12T15:29:46.634 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-07-12T15:29:46.634 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (0 ms) 2023-07-12T15:29:46.634 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-07-12T15:29:46.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2023-07-12T15:29:46.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-07-12T15:29:46.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (0 ms) 2023-07-12T15:29:46.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-07-12T15:29:46.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2 ms) 2023-07-12T15:29:46.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-07-12T15:29:46.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1 ms) 2023-07-12T15:29:46.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-07-12T15:29:46.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1 ms) 2023-07-12T15:29:46.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] 12 tests from AsioRados (26 ms total) 2023-07-12T15:29:46.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2023-07-12T15:29:46.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] Global test environment tear-down 2023-07-12T15:29:46.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (5947 ms total) 2023-07-12T15:29:46.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ PASSED ] 12 tests. 2023-07-12T15:29:46.832 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2023-07-12T15:29:46.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] Global test environment set-up. 2023-07-12T15:29:46.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 7 tests from LibRadosList 2023-07-12T15:29:46.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-07-12T15:29:46.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjects (84 ms) 2023-07-12T15:29:46.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-07-12T15:29:46.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (17 ms) 2023-07-12T15:29:46.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-07-12T15:29:46.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo2,foo3 2023-07-12T15:29:46.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2023-07-12T15:29:46.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo2 2023-07-12T15:29:46.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo3 2023-07-12T15:29:46.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo4,foo5 2023-07-12T15:29:46.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo4 2023-07-12T15:29:46.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo5 2023-07-12T15:29:46.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2023-07-12T15:29:46.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo6,foo7 2023-07-12T15:29:46.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo7 2023-07-12T15:29:46.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo6 2023-07-12T15:29:46.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-07-12T15:29:46.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo4 2023-07-12T15:29:46.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo5 2023-07-12T15:29:46.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo7 2023-07-12T15:29:46.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo6 2023-07-12T15:29:46.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo1 2023-07-12T15:29:46.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo1 2023-07-12T15:29:46.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo2 2023-07-12T15:29:46.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo3 2023-07-12T15:29:46.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (53 ms) 2023-07-12T15:29:46.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-07-12T15:29:46.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 1 0 2023-07-12T15:29:46.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 10 0 2023-07-12T15:29:46.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 13 0 2023-07-12T15:29:46.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 7 0 2023-07-12T15:29:46.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 14 0 2023-07-12T15:29:46.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 0 0 2023-07-12T15:29:46.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 15 0 2023-07-12T15:29:46.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 11 0 2023-07-12T15:29:46.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 5 0 2023-07-12T15:29:46.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 8 0 2023-07-12T15:29:46.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 6 0 2023-07-12T15:29:46.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 3 0 2023-07-12T15:29:46.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 4 0 2023-07-12T15:29:46.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 12 0 2023-07-12T15:29:46.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 9 0 2023-07-12T15:29:46.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2 0 2023-07-12T15:29:46.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-07-12T15:29:46.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (53 ms) 2023-07-12T15:29:46.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-07-12T15:29:46.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: x cursor=MIN 2023-07-12T15:29:46.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=1 cursor=9:02547ec2:::1:head 2023-07-12T15:29:46.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=10 cursor=9:52ea6a34:::10:head 2023-07-12T15:29:46.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=13 cursor=9:566253c9:::13:head 2023-07-12T15:29:46.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=7 cursor=9:5c6b0b28:::7:head 2023-07-12T15:29:46.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=14 cursor=9:62a1935d:::14:head 2023-07-12T15:29:46.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=0 cursor=9:6cac518f:::0:head 2023-07-12T15:29:46.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=15 cursor=9:863748b0:::15:head 2023-07-12T15:29:46.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=11 cursor=9:89d3ae78:::11:head 2023-07-12T15:29:46.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=5 cursor=9:b29083e3:::5:head 2023-07-12T15:29:46.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=8 cursor=9:bd63b0f1:::8:head 2023-07-12T15:29:46.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=6 cursor=9:c4fdafeb:::6:head 2023-07-12T15:29:46.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=3 cursor=9:cfc208b3:::3:head 2023-07-12T15:29:46.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=4 cursor=9:d83876eb:::4:head 2023-07-12T15:29:46.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=12 cursor=9:de5d7c5f:::12:head 2023-07-12T15:29:46.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=9 cursor=9:e960b815:::9:head 2023-07-12T15:29:46.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=2 cursor=9:f905c69b:::2:head 2023-07-12T15:29:46.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: FIRST> seek to MIN oid=1 2023-07-12T15:29:46.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=1 cursor=9:02547ec2:::1:head 2023-07-12T15:29:46.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:02547ec2:::1:head 2023-07-12T15:29:46.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:02547ec2:::1:head -> 1 2023-07-12T15:29:46.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=10 cursor=9:52ea6a34:::10:head 2023-07-12T15:29:46.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:52ea6a34:::10:head 2023-07-12T15:29:46.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2023-07-12T15:29:46.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=13 cursor=9:566253c9:::13:head 2023-07-12T15:29:46.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:566253c9:::13:head 2023-07-12T15:29:46.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:566253c9:::13:head -> 13 2023-07-12T15:29:46.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=7 cursor=9:5c6b0b28:::7:head 2023-07-12T15:29:46.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2023-07-12T15:29:46.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2023-07-12T15:29:46.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=14 cursor=9:62a1935d:::14:head 2023-07-12T15:29:46.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:46 smithi145 ceph-mon[95801]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:46.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2167334903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-133924-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2750524528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-133790-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:46 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:29:46.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:46 smithi145 ceph-mon[95801]: osdmap e73: 8 total, 8 up, 8 in 2023-07-12T15:29:46.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:46.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/128857192' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-133643-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-133627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-133635-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-133619-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-133656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.899 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:62a1935d:::14:head 2023-07-12T15:29:46.900 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:62a1935d:::14:head -> 14 2023-07-12T15:29:46.900 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=0 cursor=9:6cac518f:::0:head 2023-07-12T15:29:46.900 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:6cac518f:::0:head 2023-07-12T15:29:46.900 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:6cac518f:::0:head -> 0 2023-07-12T15:29:46.900 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=15 cursor=9:863748b0:::15:head 2023-07-12T15:29:46.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:863748b0:::15:head 2023-07-12T15:29:46.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:863748b0:::15:head -> 15 2023-07-12T15:29:46.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=11 cursor=9:89d3ae78:::11:head 2023-07-12T15:29:46.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:89d3ae78:::11:head 2023-07-12T15:29:46.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2023-07-12T15:29:46.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=5 cursor=9:b29083e3:::5:head 2023-07-12T15:29:46.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:b29083e3:::5:head 2023-07-12T15:29:46.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:b29083e3:::5:head -> 5 2023-07-12T15:29:46.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=8 cursor=9:bd63b0f1:::8:head 2023-07-12T15:29:46.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2023-07-12T15:29:46.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2023-07-12T15:29:46.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=6 cursor=9:c4fdafeb:::6:head 2023-07-12T15:29:46.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2023-07-12T15:29:46.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2023-07-12T15:29:46.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=3 cursor=9:cfc208b3:::3:head 2023-07-12T15:29:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:cfc208b3:::3:head 2023-07-12T15:29:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2023-07-12T15:29:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=4 cursor=9:d83876eb:::4:head 2023-07-12T15:29:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:d83876eb:::4:head 2023-07-12T15:29:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:d83876eb:::4:head -> 4 2023-07-12T15:29:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=12 cursor=9:de5d7c5f:::12:head 2023-07-12T15:29:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2023-07-12T15:29:46.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2023-07-12T15:29:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=9 cursor=9:e960b815:::9:head 2023-07-12T15:29:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:e960b815:::9:head 2023-07-12T15:29:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:e960b815:::9:head -> 9 2023-07-12T15:29:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=2 cursor=9:f905c69b:::2:head 2023-07-12T15:29:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:f905c69b:::2:head 2023-07-12T15:29:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:f905c69b:::2:head -> 2 2023-07-12T15:29:46.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:52ea6a34:::10:head 2023-07-12T15:29:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:52ea6a34:::10:head expected=9:52ea6a34:::10:head 2023-07-12T15:29:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2023-07-12T15:29:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=10 expected=10 2023-07-12T15:29:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:02547ec2:::1:head 2023-07-12T15:29:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:02547ec2:::1:head expected=9:02547ec2:::1:head 2023-07-12T15:29:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:02547ec2:::1:head -> 1 2023-07-12T15:29:46.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=1 expected=1 2023-07-12T15:29:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:d83876eb:::4:head 2023-07-12T15:29:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:d83876eb:::4:head expected=9:d83876eb:::4:head 2023-07-12T15:29:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:d83876eb:::4:head -> 4 2023-07-12T15:29:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=4 expected=4 2023-07-12T15:29:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:89d3ae78:::11:head 2023-07-12T15:29:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:89d3ae78:::11:head expected=9:89d3ae78:::11:head 2023-07-12T15:29:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2023-07-12T15:29:46.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=11 expected=11 2023-07-12T15:29:46.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:f905c69b:::2:head 2023-07-12T15:29:46.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:f905c69b:::2:head expected=9:f905c69b:::2:head 2023-07-12T15:29:46.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:f905c69b:::2:head -> 2 2023-07-12T15:29:46.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=2 expected=2 2023-07-12T15:29:46.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:e960b815:::9:head 2023-07-12T15:29:46.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:e960b815:::9:head expected=9:e960b815:::9:head 2023-07-12T15:29:46.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:e960b815:::9:head -> 9 2023-07-12T15:29:46.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=9 expected=9 2023-07-12T15:29:46.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2023-07-12T15:29:46.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:de5d7c5f:::12:head expected=9:de5d7c5f:::12:head 2023-07-12T15:29:46.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2023-07-12T15:29:46.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=12 expected=12 2023-07-12T15:29:46.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:863748b0:::15:head 2023-07-12T15:29:46.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:863748b0:::15:head expected=9:863748b0:::15:head 2023-07-12T15:29:46.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:863748b0:::15:head -> 15 2023-07-12T15:29:46.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=15 expected=15 2023-07-12T15:29:46.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:6cac518f:::0:head 2023-07-12T15:29:46.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[102777]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:46.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2167334903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-133924-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2750524528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-133790-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:29:46.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[102777]: osdmap e73: 8 total, 8 up, 8 in 2023-07-12T15:29:46.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:46.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/128857192' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-133643-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-133627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-133635-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-133619-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-133656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[94241]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:46.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2167334903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-133924-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2750524528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-133790-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:29:46.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[94241]: osdmap e73: 8 total, 8 up, 8 in 2023-07-12T15:29:46.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:46.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/128857192' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-133643-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-133627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-133635-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-133619-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:46.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-133656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: pgmap v52: 420 pgs: 288 unknown, 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:29:47.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:47.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1472338378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-133867-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/291168471' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-134082-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:47.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: pgmap v54: 676 pgs: 255 creating+peering, 147 active+clean, 274 unknown; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:47.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: osdmap e74: 8 total, 8 up, 8 in 2023-07-12T15:29:47.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-133924-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/878887150' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2023-07-12T15:29:47.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-133790-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1344950642' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-133717-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2450846833' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1447324765' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-133837-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2484798519' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-133932-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-133867-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3933476327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-133849-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/329928806' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/128857192' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-133643-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-133627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-133635-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-133619-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-133656-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: osdmap e75: 8 total, 8 up, 8 in 2023-07-12T15:29:47.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2526224747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133971-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3746686849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-133664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1570616817' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134092-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:47.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-134082-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133971-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:47.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-133664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.856 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134092-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.856 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:47.856 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.856 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2155004451' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2023-07-12T15:29:47.856 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:47.856 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]: dispatch 2023-07-12T15:29:47.856 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]: dispatch 2023-07-12T15:29:47.857 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.857 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.857 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.857 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:47 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:29:47.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: pgmap v52: 420 pgs: 288 unknown, 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:29:47.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:47.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1472338378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-133867-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/291168471' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-134082-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:47.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: pgmap v54: 676 pgs: 255 creating+peering, 147 active+clean, 274 unknown; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:47.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: osdmap e74: 8 total, 8 up, 8 in 2023-07-12T15:29:47.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-133924-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/878887150' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2023-07-12T15:29:47.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-133790-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1344950642' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-133717-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2450846833' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1447324765' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-133837-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2484798519' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-133932-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-133867-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3933476327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-133849-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/329928806' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/128857192' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-133643-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-133627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-133635-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-133619-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-133656-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: osdmap e75: 8 total, 8 up, 8 in 2023-07-12T15:29:47.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2526224747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133971-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3746686849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-133664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.990 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1570616817' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134092-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:47.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-134082-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133971-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:47.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-133664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134092-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:47.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2155004451' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2023-07-12T15:29:47.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:47.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]: dispatch 2023-07-12T15:29:47.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]: dispatch 2023-07-12T15:29:47.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:29:47.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: pgmap v52: 420 pgs: 288 unknown, 132 active+clean; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:29:47.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:47.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1472338378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-133867-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/291168471' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-134082-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:47.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: pgmap v54: 676 pgs: 255 creating+peering, 147 active+clean, 274 unknown; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:29:47.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: osdmap e74: 8 total, 8 up, 8 in 2023-07-12T15:29:47.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-133924-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/878887150' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2023-07-12T15:29:47.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-133790-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1344950642' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-133717-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2450846833' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1447324765' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-133837-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.996 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2484798519' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-133932-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.996 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-133867-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.996 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3933476327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-133849-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.996 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/329928806' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.996 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/128857192' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi069-133643-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.996 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi069-133627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.996 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi069-133635-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi069-133619-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi069-133656-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:47.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: osdmap e75: 8 total, 8 up, 8 in 2023-07-12T15:29:47.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2526224747' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133971-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3746686849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-133664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1570616817' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134092-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.997 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:47.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-134082-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133971-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:47.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-133664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134092-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:47.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:47.998 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2155004451' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2023-07-12T15:29:47.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:47.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]: dispatch 2023-07-12T15:29:47.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]: dispatch 2023-07-12T15:29:47.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:47.999 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:48.000 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:47 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:29:48.797 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[134127]: starting. 2023-07-12T15:29:48.797 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[134127]: creating pool ceph_test_rados_list_parallel.smithi069-134100 2023-07-12T15:29:48.798 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[134127]: created object 0... 2023-07-12T15:29:48.798 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[134127]: created object 25... 2023-07-12T15:29:48.798 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[134127]: created object 49... 2023-07-12T15:29:48.798 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[134127]: finishing. 2023-07-12T15:29:48.798 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[134127]: shutting down. 2023-07-12T15:29:48.798 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[134128]: starting. 2023-07-12T15:29:48.799 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[134128]: listing objects. 2023-07-12T15:29:48.799 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[134128]: listed object 0... 2023-07-12T15:29:48.799 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[134128]: listed object 25... 2023-07-12T15:29:48.799 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[134128]: saw 50 objects 2023-07-12T15:29:48.799 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[134128]: shutting down. 2023-07-12T15:29:48.799 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:48.799 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[135020]: starting. 2023-07-12T15:29:48.800 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[135020]: creating pool ceph_test_rados_list_parallel.smithi069-134100 2023-07-12T15:29:48.800 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[135020]: created object 0... 2023-07-12T15:29:48.800 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[135020]: created object 25... 2023-07-12T15:29:48.800 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[135020]: created object 49... 2023-07-12T15:29:48.800 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[135020]: finishing. 2023-07-12T15:29:48.801 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[135020]: shutting down. 2023-07-12T15:29:48.801 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:48.801 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[135021]: starting. 2023-07-12T15:29:48.802 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[135021]: listing objects. 2023-07-12T15:29:48.802 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[135021]: listed object 0... 2023-07-12T15:29:48.802 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[135021]: listed object 25... 2023-07-12T15:29:48.802 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[135021]: saw 43 objects 2023-07-12T15:29:48.802 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[135021]: shutting down. 2023-07-12T15:29:48.802 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:48.803 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[135022]: starting. 2023-07-12T15:29:48.803 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[135022]: removed 25 objects... 2023-07-12T15:29:48.803 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[135022]: removed half of the objects 2023-07-12T15:29:48.803 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[135022]: removed 50 objects... 2023-07-12T15:29:48.803 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[135022]: removed 50 objects 2023-07-12T15:29:48.803 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[135022]: shutting down. 2023-07-12T15:29:48.803 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:48.804 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:48.804 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[135095]: starting. 2023-07-12T15:29:48.804 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[135095]: creating pool ceph_test_rados_list_parallel.smithi069-134100 2023-07-12T15:29:48.804 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[135095]: created object 0... 2023-07-12T15:29:48.804 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[135095]: created object 25... 2023-07-12T15:29:48.804 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[135095]: created object 49... 2023-07-12T15:29:48.804 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[135095]: finishing. 2023-07-12T15:29:48.805 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[135095]: shutting down. 2023-07-12T15:29:48.805 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:48.805 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:48.805 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[135096]: starting. 2023-07-12T15:29:48.806 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[135096]: listing objects. 2023-07-12T15:29:48.806 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[135096]: listed object 0... 2023-07-12T15:29:48.806 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[135096]: listed object 25... 2023-07-12T15:29:48.807 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[135096]: listed object 50... 2023-07-12T15:29:48.807 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[135096]: saw 55 objects 2023-07-12T15:29:48.807 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[135096]: shutting down. 2023-07-12T15:29:48.808 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:48.808 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:48.808 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[135097]: starting. 2023-07-12T15:29:48.808 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[135097]: added 25 objects... 2023-07-12T15:29:48.809 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[135097]: added half of the objects 2023-07-12T15:29:48.809 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[135097]: added 50 objects... 2023-07-12T15:29:48.809 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[135097]: added 50 objects 2023-07-12T15:29:48.810 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[135097]: shutting down. 2023-07-12T15:29:48.810 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:48.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: pgmap v56: 900 pgs: 255 creating+peering, 147 active+clean, 498 unknown; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 0 op/s 2023-07-12T15:29:48.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-133924-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-133790-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1344950642' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-133717-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2450846833' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1447324765' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-133837-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2484798519' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-133932-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-133867-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3933476327' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-133849-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/329928806' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-134082-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133971-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-133664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134092-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:48.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]': finished 2023-07-12T15:29:48.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: osdmap e76: 8 total, 8 up, 8 in 2023-07-12T15:29:48.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-133963-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:48.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:48.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-133963-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:48.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4247267590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:48.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:48.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:48.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:48.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:48.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-133635-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:48.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-133635-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:48.956 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ************************** api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-07-12T15:29:48.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-07-12T15:29:48.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-07-12T15:29:48.957 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-07-12T15:29:48.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (8085 ms) 2023-07-12T15:29:48.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (8085 ms total) 2023-07-12T15:29:48.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: 2023-07-12T15:29:48.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-07-12T15:29:48.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (8085 ms total) 2023-07-12T15:29:48.958 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-07-12T15:29:48.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: Running main() from gmock_main.cc 2023-07-12T15:29:48.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-07-12T15:29:48.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-07-12T15:29:48.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-07-12T15:29:48.963 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-07-12T15:29:48.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-07-12T15:29:48.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-07-12T15:29:48.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (230 ms) 2023-07-12T15:29:48.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-07-12T15:29:48.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (33 ms) 2023-07-12T15:29:48.964 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-07-12T15:29:48.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (10 ms) 2023-07-12T15:29:48.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-07-12T15:29:48.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (27 ms) 2023-07-12T15:29:48.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-07-12T15:29:48.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (9 ms) 2023-07-12T15:29:48.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-07-12T15:29:48.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (5 ms) 2023-07-12T15:29:48.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-07-12T15:29:48.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (3 ms) 2023-07-12T15:29:48.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-07-12T15:29:48.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (7 ms) 2023-07-12T15:29:48.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-07-12T15:29:48.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (3 ms) 2023-07-12T15:29:48.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-07-12T15:29:48.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (4 ms) 2023-07-12T15:29:48.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-07-12T15:29:48.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (8 ms) 2023-07-12T15:29:48.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-07-12T15:29:48.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (8 ms) 2023-07-12T15:29:48.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-07-12T15:29:48.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (15 ms) 2023-07-12T15:29:48.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-07-12T15:29:48.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (10 ms) 2023-07-12T15:29:48.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-07-12T15:29:48.968 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (10 ms) 2023-07-12T15:29:48.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-07-12T15:29:48.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (3 ms) 2023-07-12T15:29:48.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (385 ms total) 2023-07-12T15:29:48.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: 2023-07-12T15:29:48.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-07-12T15:29:48.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (8113 ms total) 2023-07-12T15:29:48.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-07-12T15:29:48.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: Running main() from gmock_main.cc 2023-07-12T15:29:48.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-07-12T15:29:48.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-07-12T15:29:48.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-07-12T15:29:48.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-07-12T15:29:48.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (654 ms) 2023-07-12T15:29:48.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-07-12T15:29:48.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (18 ms) 2023-07-12T15:29:48.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-07-12T15:29:48.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (7475 ms) 2023-07-12T15:29:48.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (8147 ms total) 2023-07-12T15:29:48.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: 2023-07-12T15:29:48.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-07-12T15:29:48.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (8147 ms total) 2023-07-12T15:29:48.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-07-12T15:29:48.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: pgmap v56: 900 pgs: 255 creating+peering, 147 active+clean, 498 unknown; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 0 op/s 2023-07-12T15:29:48.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-133924-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-133790-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1344950642' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-133717-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2450846833' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1447324765' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-133837-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2484798519' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-133932-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-133867-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3933476327' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-133849-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/329928806' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-134082-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133971-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-133664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134092-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:48.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]': finished 2023-07-12T15:29:48.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: osdmap e76: 8 total, 8 up, 8 in 2023-07-12T15:29:48.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-133963-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:48.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:48.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-133963-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:48.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4247267590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:48.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:48.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:48.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:48.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:48.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-133635-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:48.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-133635-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:48.991 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: pgmap v56: 900 pgs: 255 creating+peering, 147 active+clean, 498 unknown; 582 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 0 op/s 2023-07-12T15:29:48.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi069-133924-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi069-133790-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1344950642' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi069-133717-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2450846833' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1447324765' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi069-133837-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2484798519' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi069-133932-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi069-133867-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3933476327' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi069-133849-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/329928806' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi069-134082-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133971-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi069-133664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134092-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:48.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:48.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]': finished 2023-07-12T15:29:48.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: osdmap e76: 8 total, 8 up, 8 in 2023-07-12T15:29:48.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-133963-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:48.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:48.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-133963-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:48.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4247267590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:48.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:48.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:48.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:48.998 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:48.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-133635-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:48.999 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-133635-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:49.092 INFO:tasks.workunit.client.0.smithi069.stdout:***** 2023-07-12T15:29:49.092 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.092 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[135163]: starting. 2023-07-12T15:29:49.093 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[135163]: creating pool ceph_test_rados_list_parallel.smithi069-134100 2023-07-12T15:29:49.093 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[135163]: created object 0... 2023-07-12T15:29:49.093 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[135163]: created object 25... 2023-07-12T15:29:49.093 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[135163]: created object 49... 2023-07-12T15:29:49.093 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[135163]: finishing. 2023-07-12T15:29:49.093 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[135163]: shutting down. 2023-07-12T15:29:49.094 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.094 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.094 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.094 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[135164]: starting. 2023-07-12T15:29:49.094 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[135164]: listing objects. 2023-07-12T15:29:49.094 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[135164]: listed object 0... 2023-07-12T15:29:49.095 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[135164]: listed object 25... 2023-07-12T15:29:49.095 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[135164]: listed object 50... 2023-07-12T15:29:49.095 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[135164]: listed object 75... 2023-07-12T15:29:49.095 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[135164]: saw 98 objects 2023-07-12T15:29:49.095 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[135164]: shutting down. 2023-07-12T15:29:49.095 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.095 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.096 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.096 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[135167]: starting. 2023-07-12T15:29:49.096 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[135167]: removed 25 objects... 2023-07-12T15:29:49.096 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[135167]: removed half of the objects 2023-07-12T15:29:49.096 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[135167]: removed 50 objects... 2023-07-12T15:29:49.096 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[135167]: removed 50 objects 2023-07-12T15:29:49.097 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[135167]: shutting down. 2023-07-12T15:29:49.097 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.097 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.097 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.097 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[135165]: starting. 2023-07-12T15:29:49.097 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[135165]: added 25 objects... 2023-07-12T15:29:49.098 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[135165]: added half of the objects 2023-07-12T15:29:49.098 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[135165]: added 50 objects... 2023-07-12T15:29:49.098 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[135165]: added 50 objects 2023-07-12T15:29:49.098 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[135165]: shutting down. 2023-07-12T15:29:49.098 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.099 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.099 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.099 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[135166]: starting. 2023-07-12T15:29:49.099 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[135166]: added 25 objects... 2023-07-12T15:29:49.100 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[135166]: added half of the objects 2023-07-12T15:29:49.100 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[135166]: added 50 objects... 2023-07-12T15:29:49.100 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[135166]: added 50 objects 2023-07-12T15:29:49.100 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[135166]: shutting down. 2023-07-12T15:29:49.101 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.101 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.101 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.101 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.102 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135257]: starting. 2023-07-12T15:29:49.102 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135257]: creating pool ceph_test_rados_list_parallel.smithi069-134100 2023-07-12T15:29:49.102 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135257]: created object 0... 2023-07-12T15:29:49.102 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135257]: created object 25... 2023-07-12T15:29:49.103 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135257]: created object 49... 2023-07-12T15:29:49.103 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135257]: finishing. 2023-07-12T15:29:49.103 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[135257]: shutting down. 2023-07-12T15:29:49.103 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.103 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.104 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.104 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:49.104 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135258]: starting. 2023-07-12T15:29:49.104 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135258]: listing objects. 2023-07-12T15:29:49.545 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: prT15:29:40.980+0000 7f4b9effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-07-12T15:29:49.545 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 5 monclient: authenticate success, global_id 14823 2023-07-12T15:29:49.545 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 10 monclient: _renew_subs 2023-07-12T15:29:49.545 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-07-12T15:29:49.545 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 1 -- 172.21.15.69:0/2979290620 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a288b813a0 con 0x55a289769af0 2023-07-12T15:29:49.546 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 10 monclient: _renew_subs 2023-07-12T15:29:49.546 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2023-07-12T15:29:49.546 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 1 -- 172.21.15.69:0/2979290620 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a288b67200 con 0x55a289769af0 2023-07-12T15:29:49.546 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 10 monclient: shutdown 2023-07-12T15:29:49.546 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 20 monclient: shutdown discarding 0 pending message(s) 2023-07-12T15:29:49.546 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 1 -- 172.21.15.69:0/2979290620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a289769af0 msgr2=0x55a2897fa250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:29:49.546 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 1 --2- 172.21.15.69:0/2979290620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a289769af0 0x55a2897fa250 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800a790 tx=0x7f4b9800a870 comp rx=0 tx=0).stop 2023-07-12T15:29:49.547 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 1 -- 172.21.15.69:0/2979290620 shutdown_connections 2023-07-12T15:29:49.547 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 1 --2- 172.21.15.69:0/2979290620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a2898006b0 0x55a2897facd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:29:49.547 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 1 --2- 172.21.15.69:0/2979290620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a289769af0 0x55a2897fa250 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800a790 tx=0x7f4b9800a870 comp rx=0 tx=0).stop 2023-07-12T15:29:49.547 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 1 --2- 172.21.15.69:0/2979290620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a28976a400 0x55a2897fa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:29:49.547 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 1 -- 172.21.15.69:0/2979290620 >> 172.21.15.69:0/2979290620 conn(0x55a289765570 msgr2=0x55a289801970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:29:49.547 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 1 -- 172.21.15.69:0/2979290620 shutdown_connections 2023-07-12T15:29:49.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:40.980+0000 7f4bb14208c0 1 -- 172.21.15.69:0/2979290620 wait complete. 2023-07-12T15:29:49.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 1 2023-07-12T15:29:49.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2023-07-12T15:29:49.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:41.012+0000 7f4bb14208c0 -1 failed for service _ceph-mon._tcp 2023-07-12T15:29:49.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:41.013+0000 7f4bb14208c0 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-07-12T15:29:49.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (301 ms) 2023-07-12T15:29:49.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscConnectFailure (301 ms total) 2023-07-12T15:29:49.549 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:29:49.549 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-07-12T15:29:49.549 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-07-12T15:29:49.549 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x55a288b55610 2023-07-12T15:29:50.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:50.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:50.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/126582223' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2023-07-12T15:29:50.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4247267590' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:50.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-133635-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:50.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: osdmap e77: 8 total, 8 up, 8 in 2023-07-12T15:29:50.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4180400554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3992434328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-133619-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-133635-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:50.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-133619-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-133635-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:50.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:50.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:50.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:50.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:50.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:50.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:50.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:50.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:50.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:50.152 INFO:tasks.workunit.client.0.smithi069.stdout: api_miocess_15_[135258]: listed object 0... 2023-07-12T15:29:50.153 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135258]: listed object 25... 2023-07-12T15:29:50.153 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135258]: listed object 50... 2023-07-12T15:29:50.153 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135258]: listed object 75... 2023-07-12T15:29:50.153 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135258]: listed object 100... 2023-07-12T15:29:50.153 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135258]: listed object 125... 2023-07-12T15:29:50.153 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135258]: saw 150 objects 2023-07-12T15:29:50.153 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[135258]: shutting down. 2023-07-12T15:29:50.154 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:50.154 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:50.154 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:50.154 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:50.154 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[135260]: starting. 2023-07-12T15:29:50.154 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[135260]: added 25 objects... 2023-07-12T15:29:50.154 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[135260]: added half of the objects 2023-07-12T15:29:50.154 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[135260]: added 50 objects... 2023-07-12T15:29:50.155 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[135260]: added 50 objects 2023-07-12T15:29:50.155 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[135260]: shutting down. 2023-07-12T15:29:50.155 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:50.155 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:50.155 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:50.155 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:50.155 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2023-07-12T15:29:50.156 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******* SUCCESS ********** 2023-07-12T15:29:50.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:50.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:50.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/126582223' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2023-07-12T15:29:50.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4247267590' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:50.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-133635-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: osdmap e77: 8 total, 8 up, 8 in 2023-07-12T15:29:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4180400554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3992434328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-133619-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-133635-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-133619-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-133635-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:50.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:50.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:50.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:50.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:50.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:50.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:50.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:50.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:50.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:50.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:50.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:50.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/126582223' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2023-07-12T15:29:50.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4247267590' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:50.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-133635-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:50.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: osdmap e77: 8 total, 8 up, 8 in 2023-07-12T15:29:50.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4180400554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3992434328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-133619-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-133635-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:50.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-133619-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-133635-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:50.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:50.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:50.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:50.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:50.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:50.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:50.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:50.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:50.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:50.426 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: Running main() from gmock_main.cc 2023-07-12T15:29:50.426 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-07-12T15:29:50.426 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] Global test environment set-up. 2023-07-12T15:29:50.426 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-07-12T15:29:50.426 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-07-12T15:29:50.426 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (5710 ms) 2023-07-12T15:29:50.427 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-07-12T15:29:50.427 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (338 ms) 2023-07-12T15:29:50.427 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-07-12T15:29:50.427 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (3563 ms) 2023-07-12T15:29:50.427 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-07-12T15:29:50.427 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:48.570137+0000 mon.a [INF] from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:50.427 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:48.948012+0000 mon.a [INF] from='client.? 172.21.15.69:0/4247267590' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:50.428 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:48.948480+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi069-133635-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:50.428 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:48.949824+0000 mon.c [INF] from='client.? 172.21.15.69:0/4180400554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.428 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:48.950721+0000 mon.b [INF] from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.428 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:48.951051+0000 mon.c [INF] from='client.? 172.21.15.69:0/3992434328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-133619-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.428 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:48.954994+0000 mon.c [INF] from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-133635-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:50.428 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:48.959908+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.429 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:48.962474+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-133619-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.429 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:48.962834+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-133635-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:50.429 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:48.963431+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:50.429 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:48.995231+0000 mon.a [INF] from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:50.916 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:48.997553+0000 mon.b [INF] from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:50.916 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:49.010147+0000 mon.b [INF] from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:50.916 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:49.274822+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:50.917 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:49.275195+0000 mon.a [INF] from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:50.917 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:49.275589+0000 mon.b [INF] from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:50.917 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:49.537853+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:50.917 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:49.538824+0000 mon.b [INF] from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:50.917 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:49.539495+0000 mon.a [INF] from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:50.917 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:49.809835+0000 mon.a [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:29:50.917 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:49.813560+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:50.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:49.813854+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:50.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:49.864811+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-133963-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]': finished 2023-07-12T15:29:50.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:49.864979+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:50.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:49.865018+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-133619-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:50.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.149589+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:50.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.149730+0000 mon.a [INF] from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:50.918 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.311188+0000 mon.c [INF] from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2023-07-12T15:29:51.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: pgmap v59: 644 pgs: 96 creating+peering, 548 active+clean; 609 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 116 KiB/s wr, 534 op/s 2023-07-12T15:29:51.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:51.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:29:51.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:51.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:51.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-133963-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]': finished 2023-07-12T15:29:51.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:51.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-133619-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:51.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:51.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:51.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2023-07-12T15:29:51.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2023-07-12T15:29:51.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2023-07-12T15:29:51.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2023-07-12T15:29:51.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2023-07-12T15:29:51.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2023-07-12T15:29:51.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2023-07-12T15:29:51.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2023-07-12T15:29:51.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2023-07-12T15:29:51.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2023-07-12T15:29:51.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: osdmap e78: 8 total, 8 up, 8 in 2023-07-12T15:29:51.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:51.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-133867-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:51.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-133643-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:51.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-07-12T15:29:51.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:51.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1542680339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:51.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4208866263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:51.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: pgmap v59: 644 pgs: 96 creating+peering, 548 active+clean; 609 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 116 KiB/s wr, 534 op/s 2023-07-12T15:29:51.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:51.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:29:51.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:51.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:51.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-133963-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]': finished 2023-07-12T15:29:51.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:51.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-133619-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:51.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:51.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:51.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2023-07-12T15:29:51.246 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2023-07-12T15:29:51.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2023-07-12T15:29:51.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2023-07-12T15:29:51.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2023-07-12T15:29:51.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2023-07-12T15:29:51.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2023-07-12T15:29:51.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2023-07-12T15:29:51.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2023-07-12T15:29:51.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2023-07-12T15:29:51.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: osdmap e78: 8 total, 8 up, 8 in 2023-07-12T15:29:51.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:51.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-133867-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:51.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-133643-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:51.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-07-12T15:29:51.249 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:51.249 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1542680339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:51.249 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4208866263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: pgmap v59: 644 pgs: 96 creating+peering, 548 active+clean; 609 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 116 KiB/s wr, 534 op/s 2023-07-12T15:29:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:29:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:51.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi069-133963-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]': finished 2023-07-12T15:29:51.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:51.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi069-133619-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:51.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:51.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:51.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2023-07-12T15:29:51.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2023-07-12T15:29:51.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2023-07-12T15:29:51.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2023-07-12T15:29:51.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2023-07-12T15:29:51.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2023-07-12T15:29:51.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2023-07-12T15:29:51.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2023-07-12T15:29:51.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2023-07-12T15:29:51.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2023-07-12T15:29:51.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: osdmap e78: 8 total, 8 up, 8 in 2023-07-12T15:29:51.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:51.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-133867-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:51.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-133643-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:51.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-07-12T15:29:51.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:51.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1542680339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:51.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4208866263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:51.914 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.311777+0000 mon.c [INF] from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2023-07-12T15:29:51.914 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.312187+0000 mon.c [INF] from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2023-07-12T15:29:51.914 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.312625+0000 mon.c [INF] from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2023-07-12T15:29:51.915 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.313025+0000 mon.c [INF] from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2023-07-12T15:29:51.915 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.313406+0000 mon.c [INF] from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2023-07-12T15:29:51.915 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.313784+0000 mon.c [INF] from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2023-07-12T15:29:51.915 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.314181+0000 mon.c [INF] from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2023-07-12T15:29:51.915 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.314563+0000 mon.c [INF] from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2023-07-12T15:29:51.915 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.314984+0000 mon.c [INF] from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2023-07-12T15:29:51.915 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.403617+0000 mon.b [INF] from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:51.916 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.403861+0000 mon.b [INF] from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-133867-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:51.916 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.404670+0000 mon.a [INF] from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-133643-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:51.916 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.424399+0000 mon.c [INF] from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-07-12T15:29:51.916 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.654269+0000 mon.a [INF] from='client.? 172.21.15.69:0/1542680339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:51.916 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.654430+0000 mon.a [INF] from='client.? 172.21.15.69:0/4208866263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:51.916 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.424592+0000 client.admin [INF] onexx 2023-07-12T15:29:51.916 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.909861+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:51.917 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.919389+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-07-12T15:29:51.996 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: go delete_pools_parallel: process_1_[134178]: starting. 2023-07-12T15:29:51.996 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[134178]: creating pool ceph_test_rados_delete_pools_parallel.smithi069-134124 2023-07-12T15:29:51.996 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[134178]: created object 0... 2023-07-12T15:29:51.996 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[134178]: created object 25... 2023-07-12T15:29:51.997 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[134178]: created object 49... 2023-07-12T15:29:51.997 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[134178]: finishing. 2023-07-12T15:29:51.997 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[134178]: shutting down. 2023-07-12T15:29:51.997 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[134179]: starting. 2023-07-12T15:29:51.997 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[134179]: deleting pool ceph_test_rados_delete_pools_parallel.smithi069-134124 2023-07-12T15:29:51.997 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[134179]: shutting down. 2023-07-12T15:29:51.997 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-07-12T15:29:51.998 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[135328]: starting. 2023-07-12T15:29:51.998 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[135328]: creating pool ceph_test_rados_delete_pools_parallel.smithi069-134124 2023-07-12T15:29:51.998 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[135328]: created object 0... 2023-07-12T15:29:51.998 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[135328]: created object 25... 2023-07-12T15:29:51.998 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[135328]: created object 49... 2023-07-12T15:29:51.998 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[135328]: finishing. 2023-07-12T15:29:51.998 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[135328]: shutting down. 2023-07-12T15:29:51.998 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-07-12T15:29:51.999 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[135330]: starting. 2023-07-12T15:29:51.999 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[135330]: listing objects. 2023-07-12T15:29:51.999 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[135330]: listed object 0... 2023-07-12T15:29:51.999 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[135330]: listed object 25... 2023-07-12T15:29:51.999 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[135330]: saw 50 objects 2023-07-12T15:29:51.999 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[135330]: shutting down. 2023-07-12T15:29:51.999 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-07-12T15:29:52.000 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[135329]: starting. 2023-07-12T15:29:52.000 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[135329]: deleting pool ceph_test_rados_delete_pools_parallel.smithi069-134124 2023-07-12T15:29:52.000 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[135329]: shutting down. 2023-07-12T15:29:52.000 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-07-12T15:29:52.000 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2023-07-12T15:29:52.000 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-07-12T15:29:52.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2023-07-12T15:29:52.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2023-07-12T15:29:52.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2023-07-12T15:29:52.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2023-07-12T15:29:52.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2023-07-12T15:29:52.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2023-07-12T15:29:52.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2023-07-12T15:29:52.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2023-07-12T15:29:52.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2023-07-12T15:29:52.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2023-07-12T15:29:52.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: 11.2 deep-scrub starts 2023-07-12T15:29:52.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: 11.2 deep-scrub ok 2023-07-12T15:29:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: onexx 2023-07-12T15:29:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: 11.8 deep-scrub starts 2023-07-12T15:29:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: 11.8 deep-scrub ok 2023-07-12T15:29:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-07-12T15:29:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-133867-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-133635-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-133635-16"}]': finished 2023-07-12T15:29:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: 11.1 deep-scrub starts 2023-07-12T15:29:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: 11.1 deep-scrub ok 2023-07-12T15:29:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1542680339' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4208866263' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: osdmap e79: 8 total, 8 up, 8 in 2023-07-12T15:29:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4180400554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: pgmap v62: 692 pgs: 208 unknown, 64 creating+peering, 420 active+clean; 586 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 21 KiB/s wr, 472 op/s 2023-07-12T15:29:52.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: 11.4 deep-scrub starts 2023-07-12T15:29:52.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: 11.4 deep-scrub ok 2023-07-12T15:29:52.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:52.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:52.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:52.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2023-07-12T15:29:52.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2023-07-12T15:29:52.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2023-07-12T15:29:52.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2023-07-12T15:29:52.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2023-07-12T15:29:52.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2023-07-12T15:29:52.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2023-07-12T15:29:52.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2023-07-12T15:29:52.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2023-07-12T15:29:52.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2023-07-12T15:29:52.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: 11.2 deep-scrub starts 2023-07-12T15:29:52.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: 11.2 deep-scrub ok 2023-07-12T15:29:52.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: onexx 2023-07-12T15:29:52.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: 11.8 deep-scrub starts 2023-07-12T15:29:52.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: 11.8 deep-scrub ok 2023-07-12T15:29:52.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-07-12T15:29:52.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-133867-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-133635-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-133635-16"}]': finished 2023-07-12T15:29:52.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: 11.1 deep-scrub starts 2023-07-12T15:29:52.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: 11.1 deep-scrub ok 2023-07-12T15:29:52.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1542680339' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:52.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4208866263' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:52.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: osdmap e79: 8 total, 8 up, 8 in 2023-07-12T15:29:52.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4180400554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: pgmap v62: 692 pgs: 208 unknown, 64 creating+peering, 420 active+clean; 586 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 21 KiB/s wr, 472 op/s 2023-07-12T15:29:52.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: 11.4 deep-scrub starts 2023-07-12T15:29:52.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: 11.4 deep-scrub ok 2023-07-12T15:29:52.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:52.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:52.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:51 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2023-07-12T15:29:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2023-07-12T15:29:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2023-07-12T15:29:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2023-07-12T15:29:52.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2023-07-12T15:29:52.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2023-07-12T15:29:52.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2023-07-12T15:29:52.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2023-07-12T15:29:52.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2023-07-12T15:29:52.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2023-07-12T15:29:52.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: 11.2 deep-scrub starts 2023-07-12T15:29:52.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: 11.2 deep-scrub ok 2023-07-12T15:29:52.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: onexx 2023-07-12T15:29:52.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: 11.8 deep-scrub starts 2023-07-12T15:29:52.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: 11.8 deep-scrub ok 2023-07-12T15:29:52.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-07-12T15:29:52.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-133867-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-133635-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-133635-16"}]': finished 2023-07-12T15:29:52.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: 11.1 deep-scrub starts 2023-07-12T15:29:52.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: 11.1 deep-scrub ok 2023-07-12T15:29:52.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1542680339' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:52.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4208866263' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:52.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: osdmap e79: 8 total, 8 up, 8 in 2023-07-12T15:29:52.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4180400554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: pgmap v62: 692 pgs: 208 unknown, 64 creating+peering, 420 active+clean; 586 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 21 KiB/s wr, 472 op/s 2023-07-12T15:29:52.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: 11.4 deep-scrub starts 2023-07-12T15:29:52.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: 11.4 deep-scrub ok 2023-07-12T15:29:52.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:52.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:52.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:51 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:52.918 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: 11.9 deep-scrub starts 2023-07-12T15:29:52.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: 11.9 deep-scrub ok 2023-07-12T15:29:52.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2023-07-12T15:29:52.919 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-07-12T15:29:52.920 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: twoxx 2023-07-12T15:29:52.920 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-07-12T15:29:52.920 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-133643-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-133643-22"}]': finished 2023-07-12T15:29:52.921 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:52.921 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-133867-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:52.921 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:52.921 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:52.922 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:52.922 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: osdmap e80: 8 total, 8 up, 8 in 2023-07-12T15:29:52.922 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-133924-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:52.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-133867-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:52.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3242928993' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-133619-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-133924-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:52.924 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-133867-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:52.924 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-133717-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.924 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-133717-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.924 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4180400554' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-133627-3", "field": "max_bytes", "val": "4096"}]: dispatch 2023-07-12T15:29:52.925 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:52.925 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-133627-3", "field": "max_bytes", "val": "4096"}]: dispatch 2023-07-12T15:29:52.925 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:52.926 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:52.926 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:52.926 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:29:52.927 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:29:52.927 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-133664-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.927 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:52.927 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:52.928 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:52.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: 11.9 deep-scrub starts 2023-07-12T15:29:52.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: 11.9 deep-scrub ok 2023-07-12T15:29:52.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2023-07-12T15:29:52.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-07-12T15:29:52.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: twoxx 2023-07-12T15:29:52.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-07-12T15:29:52.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-133643-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-133643-22"}]': finished 2023-07-12T15:29:52.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:52.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-133867-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:52.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:52.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:52.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:52.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: osdmap e80: 8 total, 8 up, 8 in 2023-07-12T15:29:52.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-133924-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:52.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-133867-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:52.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3242928993' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-133619-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:52.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-133924-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:52.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-133867-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:52.934 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-133717-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.934 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-133717-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:52.934 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4180400554' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-133627-3", "field": "max_bytes", "val": "4096"}]: dispatch 2023-07-12T15:29:52.934 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:52.935 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-133627-3", "field": "max_bytes", "val": "4096"}]: dispatch 2023-07-12T15:29:52.935 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:52.935 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:52.935 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:52.936 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:29:52.936 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:29:52.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-133664-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:52.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:52.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:52.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:52 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:53.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: Running main() from gmock_main.cc 2023-07-12T15:29:53.016 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-07-12T15:29:53.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] Global test environment set-up. 2023-07-12T15:29:53.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-07-12T15:29:53.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: seed 133643 2023-07-12T15:29:53.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-07-12T15:29:53.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2023-07-12T15:29:53.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-07-12T15:29:53.017 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (975 ms) 2023-07-12T15:29:53.018 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-07-12T15:29:53.018 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (22 ms) 2023-07-12T15:29:53.018 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-07-12T15:29:53.018 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (5 ms) 2023-07-12T15:29:53.018 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-07-12T15:29:53.018 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (4 ms) 2023-07-12T15:29:53.018 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-07-12T15:29:53.018 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (6 ms) 2023-07-12T15:29:53.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-07-12T15:29:53.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (3 ms) 2023-07-12T15:29:53.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-07-12T15:29:53.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (3 ms) 2023-07-12T15:29:53.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-07-12T15:29:53.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (6 ms) 2023-07-12T15:29:53.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-07-12T15:29:53.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (3 ms) 2023-07-12T15:29:53.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-07-12T15:29:53.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (2 ms) 2023-07-12T15:29:53.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-07-12T15:29:53.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (7 ms) 2023-07-12T15:29:53.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-07-12T15:29:53.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (5 ms) 2023-07-12T15:29:53.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-07-12T15:29:53.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (4 ms) 2023-07-12T15:29:53.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-07-12T15:29:53.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (5 ms) 2023-07-12T15:29:53.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-07-12T15:29:53.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (13 ms) 2023-07-12T15:29:53.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-07-12T15:29:53.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (4 ms) 2023-07-12T15:29:53.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-07-12T15:29:53.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (6 ms) 2023-07-12T15:29:53.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-07-12T15:29:53.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (4 ms) 2023-07-12T15:29:53.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-07-12T15:29:53.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (5 ms) 2023-07-12T15:29:53.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (1083 ms total) 2023-07-12T15:29:53.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: 2023-07-12T15:29:53.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-07-12T15:29:53.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-07-12T15:29:53.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1039 ms) 2023-07-12T15:29:53.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-07-12T15:29:53.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (11 ms) 2023-07-12T15:29:53.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-07-12T15:29:53.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (5 ms) 2023-07-12T15:29:53.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-07-12T15:29:53.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2023-07-12T15:29:53.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-07-12T15:29:53.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (4 ms) 2023-07-12T15:29:53.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: 11.9 deep-scrub starts 2023-07-12T15:29:53.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: 11.9 deep-scrub ok 2023-07-12T15:29:53.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2023-07-12T15:29:53.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-07-12T15:29:53.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: twoxx 2023-07-12T15:29:53.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-07-12T15:29:53.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi069-133643-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi069-133643-22"}]': finished 2023-07-12T15:29:53.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:53.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-133867-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:53.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:53.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:53.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:53.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: osdmap e80: 8 total, 8 up, 8 in 2023-07-12T15:29:53.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-133924-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:53.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-133867-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:53.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3242928993' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-133619-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:53.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-133924-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:53.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-133867-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:53.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-133717-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:53.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-133717-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:53.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4180400554' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-133627-3", "field": "max_bytes", "val": "4096"}]: dispatch 2023-07-12T15:29:53.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:53.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-133627-3", "field": "max_bytes", "val": "4096"}]: dispatch 2023-07-12T15:29:53.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:53.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:53.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:53.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:29:53.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:29:53.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-133664-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:53.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:53.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:53.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:52 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:53.678 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN open_pools_parallel: process_1_[134180]: starting. 2023-07-12T15:29:53.678 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[134180]: creating pool ceph_test_rados_open_pools_parallel.smithi069-134118 2023-07-12T15:29:53.679 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[134180]: created object 0... 2023-07-12T15:29:53.680 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[134180]: created object 25... 2023-07-12T15:29:53.680 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[134180]: created object 49... 2023-07-12T15:29:53.680 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[134180]: finishing. 2023-07-12T15:29:53.681 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[134180]: shutting down. 2023-07-12T15:29:53.681 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[134181]: starting. 2023-07-12T15:29:53.681 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[134181]: rados_pool_create. 2023-07-12T15:29:53.681 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[134181]: rados_ioctx_create. 2023-07-12T15:29:53.681 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[134181]: shutting down. 2023-07-12T15:29:53.681 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2023-07-12T15:29:53.681 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[135370]: starting. 2023-07-12T15:29:53.682 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[135370]: creating pool ceph_test_rados_open_pools_parallel.smithi069-134118 2023-07-12T15:29:53.682 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[135370]: created object 0... 2023-07-12T15:29:53.682 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[135370]: created object 25... 2023-07-12T15:29:53.682 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[135370]: created object 49... 2023-07-12T15:29:53.682 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[135370]: finishing. 2023-07-12T15:29:53.682 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[135370]: shutting down. 2023-07-12T15:29:53.682 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2023-07-12T15:29:53.683 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[135371]: starting. 2023-07-12T15:29:53.683 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[135371]: rados_pool_create. 2023-07-12T15:29:53.683 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[135371]: rados_ioctx_create. 2023-07-12T15:29:53.683 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[135371]: shutting down. 2023-07-12T15:29:53.683 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2023-07-12T15:29:53.683 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2023-07-12T15:29:53.683 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-07-12T15:29:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: 11.6 deep-scrub starts 2023-07-12T15:29:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: 11.6 deep-scrub ok 2023-07-12T15:29:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-07-12T15:29:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3242928993' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-133619-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:29:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:29:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: pgmap v65: 620 pgs: 200 unknown, 64 creating+peering, 356 active+clean; 586 KiB data, 261 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:29:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-133627-3", "field": "max_bytes", "val": "4096"}]': finished 2023-07-12T15:29:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]': finished 2023-07-12T15:29:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]': finished 2023-07-12T15:29:54.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-133664-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:54.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: osdmap e81: 8 total, 8 up, 8 in 2023-07-12T15:29:54.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:54.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-4", "overlaypool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:54.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:54.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1669702925' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-133664-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:29:54.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1721217336' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: 11.6 deep-scrub starts 2023-07-12T15:29:54.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: 11.6 deep-scrub ok 2023-07-12T15:29:54.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-07-12T15:29:54.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3242928993' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-133619-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:54.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:29:54.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:29:54.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: pgmap v65: 620 pgs: 200 unknown, 64 creating+peering, 356 active+clean; 586 KiB data, 261 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:29:54.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:54.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-133627-3", "field": "max_bytes", "val": "4096"}]': finished 2023-07-12T15:29:54.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]': finished 2023-07-12T15:29:54.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]': finished 2023-07-12T15:29:54.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-133664-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:54.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: osdmap e81: 8 total, 8 up, 8 in 2023-07-12T15:29:54.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:54.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-4", "overlaypool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:54.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:54.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1669702925' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-133664-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:29:54.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1721217336' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:54.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-4", "overlaypool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:54.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:54.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:54.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:54.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-4", "overlaypool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:54.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:54.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:53 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: 11.6 deep-scrub starts 2023-07-12T15:29:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: 11.6 deep-scrub ok 2023-07-12T15:29:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-07-12T15:29:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3242928993' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi069-133619-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:29:54.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:29:54.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: pgmap v65: 620 pgs: 200 unknown, 64 creating+peering, 356 active+clean; 586 KiB data, 261 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:29:54.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:54.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi069-133627-3", "field": "max_bytes", "val": "4096"}]': finished 2023-07-12T15:29:54.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi069-133963-1"}]': finished 2023-07-12T15:29:54.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]': finished 2023-07-12T15:29:54.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi069-133664-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:29:54.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: osdmap e81: 8 total, 8 up, 8 in 2023-07-12T15:29:54.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1161458702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:54.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-4", "overlaypool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:54.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1669702925' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-133664-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:29:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1721217336' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]: dispatch 2023-07-12T15:29:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-4", "overlaypool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:53 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:54.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:29:54.923 INFO:tasks.workunit.client.0.smithi069.stdout:t: 2023-07-12T15:29:50.919672+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi069-133867-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:54.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:50.928752+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi069-133635-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi069-133635-16"}]': finished 2023-07-12T15:29:54.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.208629+0000 mon.a [INF] from='client.? 172.21.15.69:0/1542680339' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:54.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.208719+0000 mon.a [INF] from='client.? 172.21.15.69:0/4208866263' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:54.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.209768+0000 mon.b [INF] from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.209942+0000 mon.c [INF] from='client.? 172.21.15.69:0/4180400554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.217859+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi069-133627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.218404+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:54.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.237905+0000 mon.c [INF] from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:54.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.476790+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:54.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.477555+0000 mon.c [INF] from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:54.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.479637+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:54.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.480143+0000 mon.c [INF] from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:54.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.480436+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:54.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.754189+0000 mon.a [INF] from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:54.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:51.762696+0000 mon.a [INF] from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:54.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:53.754851+0000 mgr.y [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-07-12T15:29:54.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:53.758291+0000 mgr.y [INF] Reconfiguring daemon alertmanager.a on smithi069 2023-07-12T15:29:54.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:53.932475+0000 mon.a [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2023-07-12T15:29:54.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:53.942994+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-133924-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-133924-7"}]': finished 2023-07-12T15:29:54.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:53.943080+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-133867-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-133867-7"}]': finished 2023-07-12T15:29:54.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:53.943141+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-133717-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-133717-10"}]': finished 2023-07-12T15:29:54.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.214769+0000 mon.a [INF] from='client.? 172.21.15.69:0/1721217336' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:54.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.214906+0000 mon.a [INF] from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:54.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.214986+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]': finished 2023-07-12T15:29:54.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.215061+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-4", "overlaypool": "test-rados-api-smithi069-133790-4-cache"}]': finished 2023-07-12T15:29:54.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.215119+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]': finished 2023-07-12T15:29:54.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.215173+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:54.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.215556+0000 mon.b [INF] from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-4-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:29:54.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.215822+0000 mon.c [INF] from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:54.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.217587+0000 mon.a [INF] from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:54.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.223306+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:54.927 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.224474+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-4-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:29:54.984 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: Running main() from gmock_main.cc 2023-07-12T15:29:54.985 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-07-12T15:29:54.985 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] Global test environment set-up. 2023-07-12T15:29:54.985 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-07-12T15:29:54.985 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-07-12T15:29:54.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (91 ms) 2023-07-12T15:29:54.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-07-12T15:29:54.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-07-12T15:29:54.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-07-12T15:29:54.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2023-07-12T15:29:54.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2023-07-12T15:29:54.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2023-07-12T15:29:54.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2023-07-12T15:29:54.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: no timeout :/ 2023-07-12T15:29:54.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (228 ms) 2023-07-12T15:29:54.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-07-12T15:29:54.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (19 ms) 2023-07-12T15:29:54.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-07-12T15:29:54.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.Checksum (5 ms) 2023-07-12T15:29:54.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-07-12T15:29:54.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (5 ms) 2023-07-12T15:29:54.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-07-12T15:29:54.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (2 ms) 2023-07-12T15:29:54.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-07-12T15:29:54.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (5 ms) 2023-07-12T15:29:54.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-07-12T15:29:54.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (1 ms) 2023-07-12T15:29:54.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-07-12T15:29:54.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.TruncTest (2 ms) 2023-07-12T15:29:54.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-07-12T15:29:54.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (3 ms) 2023-07-12T15:29:54.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-07-12T15:29:54.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (2 ms) 2023-07-12T15:29:54.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-07-12T15:29:54.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RmXattr (7 ms) 2023-07-12T15:29:54.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-07-12T15:29:54.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.XattrIter (3 ms) 2023-07-12T15:29:54.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 14 tests from LibRadosIo (374 ms total) 2023-07-12T15:29:54.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: 2023-07-12T15:29:54.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-07-12T15:29:54.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-07-12T15:29:54.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1120 ms) 2023-07-12T15:29:54.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-07-12T15:29:54.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (7 ms) 2023-07-12T15:29:54.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-07-12T15:29:54.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (4 ms) 2023-07-12T15:29:54.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-07-12T15:29:54.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (5 ms) 2023-07-12T15:29:54.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-07-12T15:29:54.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (7 ms) 2023-07-12T15:29:54.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-07-12T15:29:54.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (4 ms) 2023-07-12T15:29:54.994 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-07-12T15:29:54.994 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2023-07-12T15:29:54.994 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-07-12T15:29:54.994 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (4 ms) 2023-07-12T15:29:54.995 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-07-12T15:29:54.995 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (9 ms) 2023-07-12T15:29:54.995 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-07-12T15:29:54.995 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2023-07-12T15:29:54.995 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1167 ms total) 2023-07-12T15:29:54.995 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: 2023-07-12T15:29:54.995 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] Global test environment tear-down 2023-07-12T15:29:54.995 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [==========] 24 tests from 2 test suites ran. (14324 ms total) 2023-07-12T15:29:54.996 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ PASSED ] 24 tests. 2023-07-12T15:29:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: Reconfiguring alertmanager.a (dependencies changed)... 2023-07-12T15:29:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: Reconfiguring daemon alertmanager.a on smithi069 2023-07-12T15:29:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2023-07-12T15:29:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-133924-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-133924-7"}]': finished 2023-07-12T15:29:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-133867-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-133867-7"}]': finished 2023-07-12T15:29:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-133717-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-133717-10"}]': finished 2023-07-12T15:29:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: 11.0 deep-scrub starts 2023-07-12T15:29:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: 11.0 deep-scrub ok 2023-07-12T15:29:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1721217336' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]': finished 2023-07-12T15:29:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-4", "overlaypool": "test-rados-api-smithi069-133790-4-cache"}]': finished 2023-07-12T15:29:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]': finished 2023-07-12T15:29:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: osdmap e82: 8 total, 8 up, 8 in 2023-07-12T15:29:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-4-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:29:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:55.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:55.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:55.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-4-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:29:55.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2023-07-12T15:29:55.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:55.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:55.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:55.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: Reconfiguring alertmanager.a (dependencies changed)... 2023-07-12T15:29:55.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: Reconfiguring daemon alertmanager.a on smithi069 2023-07-12T15:29:55.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2023-07-12T15:29:55.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-133924-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-133924-7"}]': finished 2023-07-12T15:29:55.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-133867-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-133867-7"}]': finished 2023-07-12T15:29:55.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-133717-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-133717-10"}]': finished 2023-07-12T15:29:55.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: 11.0 deep-scrub starts 2023-07-12T15:29:55.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: 11.0 deep-scrub ok 2023-07-12T15:29:55.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1721217336' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:55.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:55.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]': finished 2023-07-12T15:29:55.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-4", "overlaypool": "test-rados-api-smithi069-133790-4-cache"}]': finished 2023-07-12T15:29:55.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]': finished 2023-07-12T15:29:55.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:55.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: osdmap e82: 8 total, 8 up, 8 in 2023-07-12T15:29:55.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-4-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:29:55.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:55.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:55.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:55.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-4-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:29:55.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2023-07-12T15:29:55.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:55.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:55.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: Reconfiguring alertmanager.a (dependencies changed)... 2023-07-12T15:29:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: Reconfiguring daemon alertmanager.a on smithi069 2023-07-12T15:29:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2023-07-12T15:29:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi069-133924-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi069-133924-7"}]': finished 2023-07-12T15:29:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi069-133867-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi069-133867-7"}]': finished 2023-07-12T15:29:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi069-133717-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi069-133717-10"}]': finished 2023-07-12T15:29:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: 11.0 deep-scrub starts 2023-07-12T15:29:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: 11.0 deep-scrub ok 2023-07-12T15:29:55.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1721217336' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:55.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:55.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi069-133963-1"}]': finished 2023-07-12T15:29:55.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-4", "overlaypool": "test-rados-api-smithi069-133790-4-cache"}]': finished 2023-07-12T15:29:55.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi069-133635-16"}]': finished 2023-07-12T15:29:55.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:55.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: osdmap e82: 8 total, 8 up, 8 in 2023-07-12T15:29:55.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-4-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:29:55.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1267792068' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:55.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:55.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]: dispatch 2023-07-12T15:29:55.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-4-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:29:55.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2023-07-12T15:29:55.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:55.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:29:55.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:55.884 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:55 smithi069 systemd[1]: Stopping Ceph alertmanager.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:29:55.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.235307+0000 mon.a [INF] from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:55.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.922084+0000 mon.c [INF] from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-07-12T15:29:55.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.922586+0000 client.admin [INF] threexx 2023-07-12T15:29:55.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.923006+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-07-12T15:29:55.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.939604+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:29:55.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.947632+0000 mon.a [INF] from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-133664-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-133664-10"}]': finished 2023-07-12T15:29:55.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.947797+0000 mon.a [INF] from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22"}]': finished 2023-07-12T15:29:55.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.947955+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]': finished 2023-07-12T15:29:55.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.948046+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-4-cache", "mode": "writeback"}]': finished 2023-07-12T15:29:55.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.948135+0000 mon.a [INF] from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app1","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:55.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.952064+0000 mon.a [INF] from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app2"}]: dispatch 2023-07-12T15:29:55.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.952856+0000 mon.a [INF] from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:55.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.953254+0000 mon.a [INF] from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:55.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.958345+0000 mon.c [INF] from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:55.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.959114+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:55.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:54.971389+0000 mon.a [INF] from='client.? 172.21.15.69:0/1669128393' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-133619-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:55.986 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:55.024168+0000 mon.b [INF] from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados- ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-07-12T15:29:55.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (6 ms) 2023-07-12T15:29:55.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-07-12T15:29:55.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5 ms) 2023-07-12T15:29:55.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-07-12T15:29:55.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (3 ms) 2023-07-12T15:29:55.987 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-07-12T15:29:55.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (5 ms) 2023-07-12T15:29:55.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-07-12T15:29:55.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (5 ms) 2023-07-12T15:29:55.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-07-12T15:29:55.988 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2023-07-12T15:29:55.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-07-12T15:29:55.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2023-07-12T15:29:55.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-07-12T15:29:55.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (8 ms) 2023-07-12T15:29:55.989 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-07-12T15:29:55.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2023-07-12T15:29:55.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-07-12T15:29:55.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (8 ms) 2023-07-12T15:29:55.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-07-12T15:29:55.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (4 ms) 2023-07-12T15:29:55.990 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-07-12T15:29:55.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (6 ms) 2023-07-12T15:29:55.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1123 ms total) 2023-07-12T15:29:55.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: 2023-07-12T15:29:55.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] Global test environment tear-down 2023-07-12T15:29:55.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (15330 ms total) 2023-07-12T15:29:55.991 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-07-12T15:29:56.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-07-12T15:29:56.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: threexx 2023-07-12T15:29:56.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-07-12T15:29:56.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:29:56.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-133664-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-133664-10"}]': finished 2023-07-12T15:29:56.151 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22"}]': finished 2023-07-12T15:29:56.151 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]': finished 2023-07-12T15:29:56.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-4-cache", "mode": "writeback"}]': finished 2023-07-12T15:29:56.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app1","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:56.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: osdmap e83: 8 total, 8 up, 8 in 2023-07-12T15:29:56.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app2"}]: dispatch 2023-07-12T15:29:56.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:56.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:56.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:56.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:56.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1669128393' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-133619-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:56.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-4"}]: dispatch 2023-07-12T15:29:56.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-4"}]: dispatch 2023-07-12T15:29:56.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: pgmap v68: 612 pgs: 24 creating+peering, 72 unknown, 516 active+clean; 145 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 38 MiB/s wr, 82 op/s 2023-07-12T15:29:56.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:56.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-07-12T15:29:56.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: threexx 2023-07-12T15:29:56.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-07-12T15:29:56.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:29:56.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-133664-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-133664-10"}]': finished 2023-07-12T15:29:56.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22"}]': finished 2023-07-12T15:29:56.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]': finished 2023-07-12T15:29:56.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-4-cache", "mode": "writeback"}]': finished 2023-07-12T15:29:56.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app1","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:56.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: osdmap e83: 8 total, 8 up, 8 in 2023-07-12T15:29:56.157 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app2"}]: dispatch 2023-07-12T15:29:56.157 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:56.157 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:56.157 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:56.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:56.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1669128393' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-133619-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:56.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-4"}]: dispatch 2023-07-12T15:29:56.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-4"}]: dispatch 2023-07-12T15:29:56.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: pgmap v68: 612 pgs: 24 creating+peering, 72 unknown, 516 active+clean; 145 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 38 MiB/s wr, 82 op/s 2023-07-12T15:29:56.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:56.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-07-12T15:29:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: threexx 2023-07-12T15:29:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-07-12T15:29:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:29:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi069-133664-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi069-133664-10"}]': finished 2023-07-12T15:29:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi069-133643-22"}]': finished 2023-07-12T15:29:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi069-133635-16"}]': finished 2023-07-12T15:29:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-4-cache", "mode": "writeback"}]': finished 2023-07-12T15:29:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app1","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: osdmap e83: 8 total, 8 up, 8 in 2023-07-12T15:29:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app2"}]: dispatch 2023-07-12T15:29:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-133643-22"}]: dispatch 2023-07-12T15:29:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:56.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:56.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:56.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1669128393' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-133619-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:56.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-4"}]: dispatch 2023-07-12T15:29:56.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-4"}]: dispatch 2023-07-12T15:29:56.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: pgmap v68: 612 pgs: 24 creating+peering, 72 unknown, 516 active+clean; 145 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 38 MiB/s wr, 82 op/s 2023-07-12T15:29:56.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:56.485 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[132429]: ts=2023-07-12T15:29:56.239Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2023-07-12T15:29:56.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:29:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:29:56] "GET /metrics HTTP/1.1" 200 240653 "" "Prometheus/2.43.0" 2023-07-12T15:29:56.985 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:56 smithi069 bash[136389]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a 2023-07-12T15:29:56.986 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:56 smithi069 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@alertmanager.a.service: Succeeded. 2023-07-12T15:29:56.986 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:56 smithi069 systemd[1]: Stopped Ceph alertmanager.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:29:56.986 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:56 smithi069 systemd[1]: Starting Ceph alertmanager.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:29:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2023-07-12T15:29:57.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2023-07-12T15:29:57.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: fourxx 2023-07-12T15:29:57.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2023-07-12T15:29:57.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-133643-22"}]': finished 2023-07-12T15:29:57.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app2","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1669128393' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-133619-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-4"}]': finished 2023-07-12T15:29:57.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: osdmap e84: 8 total, 8 up, 8 in 2023-07-12T15:29:57.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:57.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:57.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:57.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"dne","key":"key","value":"value"}]: dispatch 2023-07-12T15:29:57.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:57.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2023-07-12T15:29:57.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:57.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:57.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/246429711' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:57.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2077495748' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:57.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:57.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:57.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133963-6", "pg_num": 4}]: dispatch 2023-07-12T15:29:57.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133963-6", "pg_num": 4}]: dispatch 2023-07-12T15:29:57.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:29:57.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: pool 'PoolQuotaPP_smithi069-133627-3' is full (reached quota's max_bytes: 4 KiB) 2023-07-12T15:29:57.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: Health check failed: 1 pool(s) full (POOL_FULL) 2023-07-12T15:29:57.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:29:57.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]': finished 2023-07-12T15:29:57.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key1","value":"value1"}]': finished 2023-07-12T15:29:57.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]': finished 2023-07-12T15:29:57.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]': finished 2023-07-12T15:29:57.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133963-6", "pg_num": 4}]': finished 2023-07-12T15:29:57.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: osdmap e85: 8 total, 8 up, 8 in 2023-07-12T15:29:57.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:57.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:57.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2023-07-12T15:29:57.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:29:57.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:29:57.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:57.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:57.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:57.376 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:57 smithi069 podman[136543]: 2023-07-12T15:29:57.376 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2023-07-12T15:29:57.377 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2023-07-12T15:29:57.377 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: fourxx 2023-07-12T15:29:57.377 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2023-07-12T15:29:57.377 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-133643-22"}]': finished 2023-07-12T15:29:57.377 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app2","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.377 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.377 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1669128393' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-133619-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.378 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-4"}]': finished 2023-07-12T15:29:57.378 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: osdmap e84: 8 total, 8 up, 8 in 2023-07-12T15:29:57.378 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:57.378 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:57.378 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:57.378 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"dne","key":"key","value":"value"}]: dispatch 2023-07-12T15:29:57.378 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:57.379 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2023-07-12T15:29:57.379 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:57.379 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:57.379 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/246429711' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:57.379 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2077495748' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:57.379 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2023-07-12T15:29:57.380 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3871828787' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2023-07-12T15:29:57.380 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: fourxx 2023-07-12T15:29:57.380 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2023-07-12T15:29:57.380 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3877919924' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi069-133643-22"}]': finished 2023-07-12T15:29:57.380 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi069-133750-1","app": "app2","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.380 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.380 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1669128393' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi069-133619-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-4"}]': finished 2023-07-12T15:29:57.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: osdmap e84: 8 total, 8 up, 8 in 2023-07-12T15:29:57.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2890199255' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:57.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]: dispatch 2023-07-12T15:29:57.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:57.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"dne","key":"key","value":"value"}]: dispatch 2023-07-12T15:29:57.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:57.382 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2023-07-12T15:29:57.382 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:57.382 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:57.382 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/246429711' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:57.382 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2077495748' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:57.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:57.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:57.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133963-6", "pg_num": 4}]: dispatch 2023-07-12T15:29:57.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133963-6", "pg_num": 4}]: dispatch 2023-07-12T15:29:57.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:29:57.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: pool 'PoolQuotaPP_smithi069-133627-3' is full (reached quota's max_bytes: 4 KiB) 2023-07-12T15:29:57.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: Health check failed: 1 pool(s) full (POOL_FULL) 2023-07-12T15:29:57.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:29:57.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]': finished 2023-07-12T15:29:57.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key1","value":"value1"}]': finished 2023-07-12T15:29:57.384 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]': finished 2023-07-12T15:29:57.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]': finished 2023-07-12T15:29:57.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133963-6", "pg_num": 4}]': finished 2023-07-12T15:29:57.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: osdmap e85: 8 total, 8 up, 8 in 2023-07-12T15:29:57.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:57.385 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:57.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2023-07-12T15:29:57.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:29:57.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:29:57.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:57.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:57.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:57.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:57.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:57.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133963-6", "pg_num": 4}]: dispatch 2023-07-12T15:29:57.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133963-6", "pg_num": 4}]: dispatch 2023-07-12T15:29:57.387 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:29:57.388 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: pool 'PoolQuotaPP_smithi069-133627-3' is full (reached quota's max_bytes: 4 KiB) 2023-07-12T15:29:57.388 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: Health check failed: 1 pool(s) full (POOL_FULL) 2023-07-12T15:29:57.388 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:29:57.388 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-4", "tierpool": "test-rados-api-smithi069-133790-4-cache"}]': finished 2023-07-12T15:29:57.388 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key1","value":"value1"}]': finished 2023-07-12T15:29:57.389 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi069-133924-7"}]': finished 2023-07-12T15:29:57.389 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi069-133867-7"}]': finished 2023-07-12T15:29:57.389 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.389 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:57.390 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133963-6", "pg_num": 4}]': finished 2023-07-12T15:29:57.390 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: osdmap e85: 8 total, 8 up, 8 in 2023-07-12T15:29:57.390 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/388848540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:57.390 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]: dispatch 2023-07-12T15:29:57.391 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2023-07-12T15:29:57.391 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:29:57.391 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:29:57.392 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1674638849' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:57.392 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]: dispatch 2023-07-12T15:29:57.392 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:57.546 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: Running main() from gmock_main.cc 2023-07-12T15:29:57.546 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-07-12T15:29:57.546 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-07-12T15:29:57.547 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-07-12T15:29:57.547 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: seed 133924 2023-07-12T15:29:57.547 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-07-12T15:29:57.547 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (201 ms) 2023-07-12T15:29:57.547 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-07-12T15:29:57.547 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (29 ms) 2023-07-12T15:29:57.547 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-07-12T15:29:57.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2023-07-12T15:29:57.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-07-12T15:29:57.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (12 ms) 2023-07-12T15:29:57.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-07-12T15:29:57.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (34 ms) 2023-07-12T15:29:57.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (277 ms total) 2023-07-12T15:29:57.548 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: 2023-07-12T15:29:57.549 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-07-12T15:29:57.549 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-07-12T15:29:57.549 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1024 ms) 2023-07-12T15:29:57.549 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-07-12T15:29:57.549 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2023-07-12T15:29:57.549 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-07-12T15:29:57.549 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (7 ms) 2023-07-12T15:29:57.549 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-07-12T15:29:57.550 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (16 ms) 2023-07-12T15:29:57.550 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1047 ms total) 2023-07-12T15:29:57.550 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: 2023-07-12T15:29:57.550 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-07-12T15:29:57.550 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (16762 ms total) 2023-07-12T15:29:57.550 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ PASSED ] 9 tests. 2023-07-12T15:29:57.556 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: Running main() from gmock_main.cc 2023-07-12T15:29:57.557 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-07-12T15:29:57.557 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] Global test environment set-up. 2023-07-12T15:29:57.557 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-07-12T15:29:57.557 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-07-12T15:29:57.557 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.Stat (238 ms) 2023-07-12T15:29:57.557 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-07-12T15:29:57.558 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (12 ms) 2023-07-12T15:29:57.558 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-07-12T15:29:57.558 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.StatNS (14 ms) 2023-07-12T15:29:57.558 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-07-12T15:29:57.558 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2023-07-12T15:29:57.558 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-07-12T15:29:57.558 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (2 ms) 2023-07-12T15:29:57.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 5 tests from LibRadosStat (267 ms total) 2023-07-12T15:29:57.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: 2023-07-12T15:29:57.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-07-12T15:29:57.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-07-12T15:29:57.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1030 ms) 2023-07-12T15:29:57.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-07-12T15:29:57.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (11 ms) 2023-07-12T15:29:57.559 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-07-12T15:29:57.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1 ms) 2023-07-12T15:29:57.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-07-12T15:29:57.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (6 ms) 2023-07-12T15:29:57.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1048 ms total) 2023-07-12T15:29:57.560 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: 2023-07-12T15:29:57.561 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] Global test environment tear-down 2023-07-12T15:29:57.561 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (16789 ms total) 2023-07-12T15:29:57.561 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ PASSED ] 9 tests. 2023-07-12T15:29:57.735 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:29:57.374Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2023-07-12T15:29:57.736 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:29:57.374Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2023-07-12T15:29:57.736 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:29:57.375Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.69 port=9094 2023-07-12T15:29:57.736 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:29:57.376Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-07-12T15:29:57.736 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:57 smithi069 bash[136543]: 7fa10a1645737b11b9c4d5f4709ab32de76c4cab622ae1f085ab922b5f52a4c7 2023-07-12T15:29:57.736 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:29:57.399Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-12T15:29:57.736 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:29:57.400Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2023-07-12T15:29:57.737 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:29:57.402Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2023-07-12T15:29:57.737 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:57 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:29:57.402Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2023-07-12T15:29:57.737 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:57 smithi069 systemd[1]: Started Ceph alertmanager.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:29:58.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: 11.7 deep-scrub starts 2023-07-12T15:29:58.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: 11.7 deep-scrub ok 2023-07-12T15:29:58.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: 11.3 deep-scrub starts 2023-07-12T15:29:58.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: 11.3 deep-scrub ok 2023-07-12T15:29:58.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2023-07-12T15:29:58.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: pgmap v71: 600 pgs: 8 creating+peering, 108 unknown, 484 active+clean; 145 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 81 op/s 2023-07-12T15:29:58.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:58.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]': finished 2023-07-12T15:29:58.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key2","value":"value2"}]': finished 2023-07-12T15:29:58.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:29:58.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]': finished 2023-07-12T15:29:58.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: osdmap e86: 8 total, 8 up, 8 in 2023-07-12T15:29:58.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "overlaypool": "test-rados-api-smithi069-133963-6"}]: dispatch 2023-07-12T15:29:58.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1328865355' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-133619-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:58.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:58.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/985643905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-133837-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:58.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "overlaypool": "test-rados-api-smithi069-133963-6"}]: dispatch 2023-07-12T15:29:58.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key1"}]: dispatch 2023-07-12T15:29:58.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:58.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-133619-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:58.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:58.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-133837-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:58.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:58.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: 11.7 deep-scrub starts 2023-07-12T15:29:58.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: 11.7 deep-scrub ok 2023-07-12T15:29:58.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: 11.3 deep-scrub starts 2023-07-12T15:29:58.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: 11.3 deep-scrub ok 2023-07-12T15:29:58.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2023-07-12T15:29:58.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: pgmap v71: 600 pgs: 8 creating+peering, 108 unknown, 484 active+clean; 145 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 81 op/s 2023-07-12T15:29:58.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:58.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]': finished 2023-07-12T15:29:58.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key2","value":"value2"}]': finished 2023-07-12T15:29:58.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:29:58.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]': finished 2023-07-12T15:29:58.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: osdmap e86: 8 total, 8 up, 8 in 2023-07-12T15:29:58.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "overlaypool": "test-rados-api-smithi069-133963-6"}]: dispatch 2023-07-12T15:29:58.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1328865355' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-133619-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:58.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:58.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/985643905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-133837-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:58.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "overlaypool": "test-rados-api-smithi069-133963-6"}]: dispatch 2023-07-12T15:29:58.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key1"}]: dispatch 2023-07-12T15:29:58.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:58.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-133619-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:58.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:58.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-133837-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:58.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:58.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: 11.7 deep-scrub starts 2023-07-12T15:29:58.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: 11.7 deep-scrub ok 2023-07-12T15:29:58.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: 11.3 deep-scrub starts 2023-07-12T15:29:58.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: 11.3 deep-scrub ok 2023-07-12T15:29:58.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2023-07-12T15:29:58.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: pgmap v71: 600 pgs: 8 creating+peering, 108 unknown, 484 active+clean; 145 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 81 op/s 2023-07-12T15:29:58.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:58.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi069-133867-7"}]': finished 2023-07-12T15:29:58.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key2","value":"value2"}]': finished 2023-07-12T15:29:58.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:29:58.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi069-133924-7"}]': finished 2023-07-12T15:29:58.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: osdmap e86: 8 total, 8 up, 8 in 2023-07-12T15:29:58.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "overlaypool": "test-rados-api-smithi069-133963-6"}]: dispatch 2023-07-12T15:29:58.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1328865355' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-133619-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:58.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:58.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/985643905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-133837-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:58.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "overlaypool": "test-rados-api-smithi069-133963-6"}]: dispatch 2023-07-12T15:29:58.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key1"}]: dispatch 2023-07-12T15:29:58.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:58.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-133619-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:58.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:58.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-133837-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:58.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:59.098 INFO:tasks.workunit.client.0.smithi069.stdout:api-smithi069-133790-4"}]: dispatch 2023-07-12T15:29:59.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2023-07-12T15:29:55.265073+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-4"}]: dispatch 2023-07-12T15:29:59.098 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (8686 ms) 2023-07-12T15:29:59.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (18297 ms total) 2023-07-12T15:29:59.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: 2023-07-12T15:29:59.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] Global test environment tear-down 2023-07-12T15:29:59.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (18297 ms total) 2023-07-12T15:29:59.099 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ PASSED ] 4 tests. 2023-07-12T15:29:59.568 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: Running main() from gmock_main.cc 2023-07-12T15:29:59.569 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-07-12T15:29:59.569 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-07-12T15:29:59.569 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-07-12T15:29:59.569 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: seed 133717 2023-07-12T15:29:59.569 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-07-12T15:29:59.569 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (235 ms) 2023-07-12T15:29:59.569 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-07-12T15:29:59.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (24 ms) 2023-07-12T15:29:59.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-07-12T15:29:59.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1020 ms) 2023-07-12T15:29:59.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-07-12T15:29:59.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1080 ms) 2023-07-12T15:29:59.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-07-12T15:29:59.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (40 ms) 2023-07-12T15:29:59.570 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-07-12T15:29:59.571 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (5 ms) 2023-07-12T15:29:59.571 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-07-12T15:29:59.571 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (19 ms) 2023-07-12T15:29:59.571 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-07-12T15:29:59.571 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2023-07-12T15:29:59.571 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2426 ms total) 2023-07-12T15:29:59.571 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: 2023-07-12T15:29:59.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-07-12T15:29:59.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-07-12T15:29:59.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1065 ms) 2023-07-12T15:29:59.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-07-12T15:29:59.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (5 ms) 2023-07-12T15:29:59.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-07-12T15:29:59.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1007 ms) 2023-07-12T15:29:59.572 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-07-12T15:29:59.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1004 ms) 2023-07-12T15:29:59.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-07-12T15:29:59.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (4 ms) 2023-07-12T15:29:59.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-07-12T15:29:59.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (3 ms) 2023-07-12T15:29:59.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-07-12T15:29:59.573 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4 ms) 2023-07-12T15:29:59.574 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-07-12T15:29:59.574 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2023-07-12T15:29:59.574 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3095 ms total) 2023-07-12T15:29:59.574 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: 2023-07-12T15:29:59.574 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-07-12T15:29:59.574 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (18875 ms total) 2023-07-12T15:29:59.574 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-07-12T15:29:59.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: Reconfiguring prometheus.a (dependencies changed)... 2023-07-12T15:29:59.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: Reconfiguring daemon prometheus.a on smithi145 2023-07-12T15:29:59.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:59.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:29:59.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: 11.5 deep-scrub starts 2023-07-12T15:29:59.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "overlaypool": "test-rados-api-smithi069-133963-6"}]': finished 2023-07-12T15:29:59.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key1"}]': finished 2023-07-12T15:29:59.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-133619-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:59.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]': finished 2023-07-12T15:29:59.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-133837-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:59.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: osdmap e87: 8 total, 8 up, 8 in 2023-07-12T15:29:59.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:29:59.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133963-6", "mode": "writeback"}]: dispatch 2023-07-12T15:29:59.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133963-6", "mode": "writeback"}]: dispatch 2023-07-12T15:29:59.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4107772282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:59.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:59.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/908547232' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:59.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:59.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:59.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:59.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: 11.5 deep-scrub ok 2023-07-12T15:29:59.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:29:59.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:29:59.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:29:59.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:59.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:29:59.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:59.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:59.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: Reconfiguring prometheus.a (dependencies changed)... 2023-07-12T15:29:59.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: Reconfiguring daemon prometheus.a on smithi145 2023-07-12T15:29:59.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:59.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:29:59.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: 11.5 deep-scrub starts 2023-07-12T15:29:59.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "overlaypool": "test-rados-api-smithi069-133963-6"}]': finished 2023-07-12T15:29:59.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key1"}]': finished 2023-07-12T15:29:59.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-133619-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:59.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]': finished 2023-07-12T15:29:59.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-133837-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:59.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: osdmap e87: 8 total, 8 up, 8 in 2023-07-12T15:29:59.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:29:59.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133963-6", "mode": "writeback"}]: dispatch 2023-07-12T15:29:59.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133963-6", "mode": "writeback"}]: dispatch 2023-07-12T15:29:59.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4107772282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:59.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:59.750 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/908547232' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:59.750 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:59.750 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:59.750 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:59.750 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: 11.5 deep-scrub ok 2023-07-12T15:29:59.750 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:29:59.750 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:29:59.750 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:29:59.751 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:59.751 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:29:59.751 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:59.751 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:59.751 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:29:59 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:29:59.376Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000189104s 2023-07-12T15:29:59.757 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: Reconfiguring prometheus.a (dependencies changed)... 2023-07-12T15:29:59.758 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: Reconfiguring daemon prometheus.a on smithi145 2023-07-12T15:29:59.758 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:29:59.758 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:29:59.758 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: 11.5 deep-scrub starts 2023-07-12T15:29:59.758 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "overlaypool": "test-rados-api-smithi069-133963-6"}]': finished 2023-07-12T15:29:59.758 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2188759347' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi069-133750-1","app":"app1","key":"key1"}]': finished 2023-07-12T15:29:59.759 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi069-133619-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:59.759 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi069-133717-10"}]': finished 2023-07-12T15:29:59.759 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi069-133837-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:29:59.759 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: osdmap e87: 8 total, 8 up, 8 in 2023-07-12T15:29:59.759 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:29:59.759 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133963-6", "mode": "writeback"}]: dispatch 2023-07-12T15:29:59.759 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133963-6", "mode": "writeback"}]: dispatch 2023-07-12T15:29:59.759 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4107772282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:59.760 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/340019960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:59.760 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/908547232' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:59.760 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:59.760 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]: dispatch 2023-07-12T15:29:59.760 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:29:59.760 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: 11.5 deep-scrub ok 2023-07-12T15:29:59.760 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:29:59.761 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:29:59.761 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:29:59.761 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:29:59.761 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:29:59.761 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:29:59.761 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:29:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:00.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: pgmap v74: 552 pgs: 164 unknown, 388 active+clean; 145 MiB data, 697 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:30:00.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:30:00.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-133664-10"}]': finished 2023-07-12T15:30:00.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133963-6", "mode": "writeback"}]': finished 2023-07-12T15:30:00.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:00.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]': finished 2023-07-12T15:30:00.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:00.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:00.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: osdmap e88: 8 total, 8 up, 8 in 2023-07-12T15:30:00.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-133932-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:00.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-133932-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:00.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:30:00.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:00.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 2 stray daemon(s) not managed by cephadm; 1 pool(s) do not have an application enabled; 1 pool(s) full 2023-07-12T15:30:00.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2023-07-12T15:30:00.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: pool 'test-rados-api-smithi069-133963-6' with cache_mode writeback needs hit_set_type to be set but it is not 2023-07-12T15:30:00.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: [WRN] CEPHADM_STRAY_DAEMON: 2 stray daemon(s) not managed by cephadm 2023-07-12T15:30:00.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: stray daemon laundry.pid133993 on host smithi069 not managed by cephadm 2023-07-12T15:30:00.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: stray daemon laundry.pid134064 on host smithi069 not managed by cephadm 2023-07-12T15:30:00.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2023-07-12T15:30:00.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: application not enabled on pool 'ceph_test_rados_api_asio' 2023-07-12T15:30:00.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2023-07-12T15:30:00.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: [WRN] POOL_FULL: 1 pool(s) full 2023-07-12T15:30:00.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[102777]: pool 'PoolQuotaPP_smithi069-133627-3' is full (running out of quota) 2023-07-12T15:30:00.742 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: pgmap v74: 552 pgs: 164 unknown, 388 active+clean; 145 MiB data, 697 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:30:00.742 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:30:00.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-133664-10"}]': finished 2023-07-12T15:30:00.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133963-6", "mode": "writeback"}]': finished 2023-07-12T15:30:00.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:00.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]': finished 2023-07-12T15:30:00.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:00.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:00.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: osdmap e88: 8 total, 8 up, 8 in 2023-07-12T15:30:00.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-133932-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:00.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-133932-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:00.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:30:00.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:00.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 2 stray daemon(s) not managed by cephadm; 1 pool(s) do not have an application enabled; 1 pool(s) full 2023-07-12T15:30:00.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2023-07-12T15:30:00.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: pool 'test-rados-api-smithi069-133963-6' with cache_mode writeback needs hit_set_type to be set but it is not 2023-07-12T15:30:00.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: [WRN] CEPHADM_STRAY_DAEMON: 2 stray daemon(s) not managed by cephadm 2023-07-12T15:30:00.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: stray daemon laundry.pid133993 on host smithi069 not managed by cephadm 2023-07-12T15:30:00.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: stray daemon laundry.pid134064 on host smithi069 not managed by cephadm 2023-07-12T15:30:00.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2023-07-12T15:30:00.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: application not enabled on pool 'ceph_test_rados_api_asio' 2023-07-12T15:30:00.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2023-07-12T15:30:00.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: [WRN] POOL_FULL: 1 pool(s) full 2023-07-12T15:30:00.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:00 smithi069 ceph-mon[94241]: pool 'PoolQuotaPP_smithi069-133627-3' is full (running out of quota) 2023-07-12T15:30:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: pgmap v74: 552 pgs: 164 unknown, 388 active+clean; 145 MiB data, 697 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:30:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:30:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi069-133664-10"}]': finished 2023-07-12T15:30:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133963-6", "mode": "writeback"}]': finished 2023-07-12T15:30:00.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:00.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi069-133717-10"}]': finished 2023-07-12T15:30:00.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:00.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:00.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: osdmap e88: 8 total, 8 up, 8 in 2023-07-12T15:30:00.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-133932-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:00.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-133932-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:00.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-133664-10"}]: dispatch 2023-07-12T15:30:00.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:00.854 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 2 stray daemon(s) not managed by cephadm; 1 pool(s) do not have an application enabled; 1 pool(s) full 2023-07-12T15:30:00.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2023-07-12T15:30:00.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: pool 'test-rados-api-smithi069-133963-6' with cache_mode writeback needs hit_set_type to be set but it is not 2023-07-12T15:30:00.855 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: [WRN] CEPHADM_STRAY_DAEMON: 2 stray daemon(s) not managed by cephadm 2023-07-12T15:30:00.856 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: stray daemon laundry.pid133993 on host smithi069 not managed by cephadm 2023-07-12T15:30:00.856 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: stray daemon laundry.pid134064 on host smithi069 not managed by cephadm 2023-07-12T15:30:00.856 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2023-07-12T15:30:00.856 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: application not enabled on pool 'ceph_test_rados_api_asio' 2023-07-12T15:30:00.857 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2023-07-12T15:30:00.857 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: [WRN] POOL_FULL: 1 pool(s) full 2023-07-12T15:30:00.857 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:00 smithi145 ceph-mon[95801]: pool 'PoolQuotaPP_smithi069-133627-3' is full (running out of quota) 2023-07-12T15:30:00.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: Running main() from gmock_main.cc 2023-07-12T15:30:00.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-07-12T15:30:00.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] Global test environment set-up. 2023-07-12T15:30:00.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-07-12T15:30:00.934 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-07-12T15:30:00.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (237 ms) 2023-07-12T15:30:00.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-07-12T15:30:00.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockShared (40 ms) 2023-07-12T15:30:00.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-07-12T15:30:00.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1010 ms) 2023-07-12T15:30:00.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-07-12T15:30:00.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1059 ms) 2023-07-12T15:30:00.935 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-07-12T15:30:00.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (67 ms) 2023-07-12T15:30:00.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-07-12T15:30:00.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.Unlock (10 ms) 2023-07-12T15:30:00.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-07-12T15:30:00.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (7 ms) 2023-07-12T15:30:00.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-07-12T15:30:00.936 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (6 ms) 2023-07-12T15:30:00.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLock (2436 ms total) 2023-07-12T15:30:00.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: 2023-07-12T15:30:00.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-07-12T15:30:00.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-07-12T15:30:00.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1061 ms) 2023-07-12T15:30:00.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-07-12T15:30:00.937 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (3 ms) 2023-07-12T15:30:00.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-07-12T15:30:00.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1005 ms) 2023-07-12T15:30:00.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-07-12T15:30:00.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1006 ms) 2023-07-12T15:30:00.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-07-12T15:30:00.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (4 ms) 2023-07-12T15:30:00.938 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-07-12T15:30:00.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4 ms) 2023-07-12T15:30:00.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-07-12T15:30:00.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (4 ms) 2023-07-12T15:30:00.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-07-12T15:30:00.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3 ms) 2023-07-12T15:30:00.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3090 ms total) 2023-07-12T15:30:00.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: 2023-07-12T15:30:00.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] Global test environment tear-down 2023-07-12T15:30:00.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (20254 ms total) 2023-07-12T15:30:00.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ PASSED ] 16 tests. 2023-07-12T15:30:02.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:02.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-133664-10"}]': finished 2023-07-12T15:30:02.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:01 smithi145 ceph-mon[95801]: osdmap e89: 8 total, 8 up, 8 in 2023-07-12T15:30:02.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2605995830' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-133619-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:02.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:01 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-133619-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:02.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:02.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:02.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-133664-10"}]': finished 2023-07-12T15:30:02.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:01 smithi069 ceph-mon[94241]: osdmap e89: 8 total, 8 up, 8 in 2023-07-12T15:30:02.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2605995830' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-133619-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:02.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:01 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-133619-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:02.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:02.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:02.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/486016680' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi069-133664-10"}]': finished 2023-07-12T15:30:02.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:01 smithi069 ceph-mon[102777]: osdmap e89: 8 total, 8 up, 8 in 2023-07-12T15:30:02.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2605995830' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-133619-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:02.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:01 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-133619-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:02.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:02.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2023-07-12T15:30:02.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-07-12T15:30:02.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-07-12T15:30:02.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-07-12T15:30:02.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-07-12T15:30:02.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2023-07-12T15:30:02.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1132 ms) 2023-07-12T15:30:02.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-07-12T15:30:02.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (5 ms) 2023-07-12T15:30:02.761 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1137 ms total) 2023-07-12T15:30:02.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: 2023-07-12T15:30:02.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-07-12T15:30:02.762 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-07-12T15:30:02.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2023-07-12T15:30:02.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (51 ms) 2023-07-12T15:30:02.763 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-07-12T15:30:02.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2023-07-12T15:30:02.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3585 ms) 2023-07-12T15:30:02.764 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-07-12T15:30:02.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (4 ms) 2023-07-12T15:30:02.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-07-12T15:30:02.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (11 ms) 2023-07-12T15:30:02.765 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-07-12T15:30:02.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94838996518112 notify_id 377957122049 notifier_gid 25124 2023-07-12T15:30:02.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (9 ms) 2023-07-12T15:30:02.766 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-07-12T15:30:02.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94838996521344 notify_id 377957122050 notifier_gid 25124 2023-07-12T15:30:02.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (10 ms) 2023-07-12T15:30:02.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-07-12T15:30:02.767 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94838996521344 notify_id 377957122050 notifier_gid 25124 2023-07-12T15:30:02.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (32 ms) 2023-07-12T15:30:02.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-07-12T15:30:02.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94838996521344 notify_id 377957122051 notifier_gid 25124 2023-07-12T15:30:02.768 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (8 ms) 2023-07-12T15:30:02.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-07-12T15:30:02.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94838996518112 notify_id 377957122051 notifier_gid 25124 2023-07-12T15:30:02.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (22 ms) 2023-07-12T15:30:02.769 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-07-12T15:30:02.770 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94838996518112 notify_id 377957122052 notifier_gid 25124 2023-07-12T15:30:02.770 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (33 ms) 2023-07-12T15:30:02.770 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-07-12T15:30:02.770 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: trying... 2023-07-12T15:30:02.771 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94838996518112 notify_id 377957122052 notifier_gid 25124 2023-07-12T15:30:02.771 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: timed out 2023-07-12T15:30:02.771 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2023-07-12T15:30:03.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:02 smithi145 ceph-mon[95801]: pgmap v77: 456 pgs: 64 unknown, 392 active+clean; 145 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2023-07-12T15:30:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:02 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-133932-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]': finished 2023-07-12T15:30:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:02 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-133619-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1957919418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:03.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3685807035' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-133849-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:03.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:02 smithi145 ceph-mon[95801]: osdmap e90: 8 total, 8 up, 8 in 2023-07-12T15:30:03.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:02 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:03.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:02 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-133849-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:03.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:03.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[102777]: pgmap v77: 456 pgs: 64 unknown, 392 active+clean; 145 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2023-07-12T15:30:03.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:03.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-133932-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]': finished 2023-07-12T15:30:03.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-133619-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:03.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:03.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1957919418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3685807035' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-133849-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[102777]: osdmap e90: 8 total, 8 up, 8 in 2023-07-12T15:30:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-133849-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:03.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[94241]: pgmap v77: 456 pgs: 64 unknown, 392 active+clean; 145 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2023-07-12T15:30:03.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:03.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi069-133932-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]': finished 2023-07-12T15:30:03.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi069-133619-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:03.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:03.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1957919418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:03.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3685807035' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-133849-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:03.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[94241]: osdmap e90: 8 total, 8 up, 8 in 2023-07-12T15:30:03.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:03.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-133849-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:03.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:04.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:03 smithi145 ceph-mon[95801]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2023-07-12T15:30:04.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:03 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:04.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:03 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-133849-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:04.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:03 smithi145 ceph-mon[95801]: osdmap e91: 8 total, 8 up, 8 in 2023-07-12T15:30:04.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:04.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1008816667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:04.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:03 smithi145 ceph-mon[95801]: pgmap v80: 528 pgs: 168 unknown, 360 active+clean; 145 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2023-07-12T15:30:04.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:04.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[94241]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2023-07-12T15:30:04.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:04.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-133849-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:04.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[94241]: osdmap e91: 8 total, 8 up, 8 in 2023-07-12T15:30:04.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:04.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1008816667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:04.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[94241]: pgmap v80: 528 pgs: 168 unknown, 360 active+clean; 145 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2023-07-12T15:30:04.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:04.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[102777]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2023-07-12T15:30:04.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:04.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi069-133849-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:04.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[102777]: osdmap e91: 8 total, 8 up, 8 in 2023-07-12T15:30:04.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:04.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1008816667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:04.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[102777]: pgmap v80: 528 pgs: 168 unknown, 360 active+clean; 145 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2023-07-12T15:30:04.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:04.837 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:04.838 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1008816667' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:04.838 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:04 smithi145 ceph-mon[95801]: osdmap e92: 8 total, 8 up, 8 in 2023-07-12T15:30:04.838 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3920141121' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-133619-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:04.838 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:04.839 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:04 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:04.839 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:30:04 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:30:04.840 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:04 smithi145 systemd[1]: Stopping Ceph prometheus.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:30:05.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-7", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:30:05.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:04 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:05.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:05.102 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:30:05.047Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2023-07-12T15:30:05.102 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:30:05.047Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2023-07-12T15:30:05.102 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:30:05.047Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2023-07-12T15:30:05.102 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:30:05.047Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2023-07-12T15:30:05.103 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:30:05.047Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2023-07-12T15:30:05.103 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:30:05.047Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2023-07-12T15:30:05.103 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:30:05.047Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2023-07-12T15:30:05.103 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:30:05.047Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2023-07-12T15:30:05.103 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:30:05.047Z caller=main.go:907 level=info msg="Scrape manager stopped" 2023-07-12T15:30:05.104 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:30:05.047Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2023-07-12T15:30:05.104 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:30:05.047Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2023-07-12T15:30:05.104 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[120607]: ts=2023-07-12T15:30:05.048Z caller=main.go:1147 level=info msg="See you next time!" 2023-07-12T15:30:05.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1008816667' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[102777]: osdmap e92: 8 total, 8 up, 8 in 2023-07-12T15:30:05.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3920141121' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-133619-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:05.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:05.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:05.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-7", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:30:05.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:05.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:05.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:05.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1008816667' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:05.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[94241]: osdmap e92: 8 total, 8 up, 8 in 2023-07-12T15:30:05.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3920141121' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-133619-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:05.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:05.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:05.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-7", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:30:05.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:05.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:05.417 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 bash[124161]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a 2023-07-12T15:30:05.890 INFO:tasks.workunit.client.0.smithi069.stdout: api_ api_watch_notify: Running main() from gmock_main.cc 2023-07-12T15:30:05.891 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-07-12T15:30:05.891 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-07-12T15:30:05.891 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-07-12T15:30:05.891 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-07-12T15:30:05.891 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify_test_cb 2023-07-12T15:30:05.891 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (230 ms) 2023-07-12T15:30:05.891 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-07-12T15:30:05.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-07-12T15:30:05.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94007584269184 err -107 2023-07-12T15:30:05.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1020 ms) 2023-07-12T15:30:05.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-07-12T15:30:05.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-07-12T15:30:05.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94007584280752 err -107 2023-07-12T15:30:05.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1007 ms) 2023-07-12T15:30:05.892 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-07-12T15:30:05.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 14850 notify_id 330712481792 cookie 94007584298576 2023-07-12T15:30:05.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (9 ms) 2023-07-12T15:30:05.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-07-12T15:30:05.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 14850 notify_id 330712481792 cookie 94007584351664 2023-07-12T15:30:05.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (35 ms) 2023-07-12T15:30:05.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-07-12T15:30:05.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 14850 notify_id 335007449090 cookie 94007584351664 2023-07-12T15:30:05.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (95 ms) 2023-07-12T15:30:05.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-07-12T15:30:05.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 14850 notify_id 335007449088 cookie 94007584332576 2023-07-12T15:30:05.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 14850 notify_id 335007449088 cookie 94007584351664 2023-07-12T15:30:05.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (23 ms) 2023-07-12T15:30:05.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-07-12T15:30:05.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 14850 notify_id 335007449088 cookie 94007584332576 2023-07-12T15:30:05.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 14850 notify_id 339302416385 cookie 94007584332576 2023-07-12T15:30:05.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3023 ms) 2023-07-12T15:30:05.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-07-12T15:30:05.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-07-12T15:30:05.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94007584351664 err -107 2023-07-12T15:30:05.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 14850 notify_id 365072220163 cookie 94007584351664 2023-07-12T15:30:05.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (4009 ms) 2023-07-12T15:30:05.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-07-12T15:30:05.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-07-12T15:30:05.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94007584351664 err -107 2023-07-12T15:30:05.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1004 ms) 2023-07-12T15:30:05.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10455 ms total) 2023-07-12T15:30:05.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: 2023-07-12T15:30:05.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-07-12T15:30:05.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-07-12T15:30:05.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify_test_cb 2023-07-12T15:30:05.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1107 ms) 2023-07-12T15:30:05.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1107 ms total) 2023-07-12T15:30:05.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: 2023-07-12T15:30:05.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-07-12T15:30:05.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (25103 ms total) 2023-07-12T15:30:05.898 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ PASSED ] 11 tests. 2023-07-12T15:30:05.932 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@prometheus.a.service: Succeeded. 2023-07-12T15:30:05.933 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 systemd[1]: Stopped Ceph prometheus.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:30:05.933 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 systemd[1]: Starting Ceph prometheus.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:30:06.183 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:05 smithi145 podman[124279]: 2023-07-12T15:30:06.183 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.180Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2023-07-12T15:30:06.183 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.180Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2023-07-12T15:30:06.184 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.180Z caller=main.go:570 level=info host_details="(Linux 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 x86_64 smithi145 (none))" 2023-07-12T15:30:06.184 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.180Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2023-07-12T15:30:06.184 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.180Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2023-07-12T15:30:06.184 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.182Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2023-07-12T15:30:06.184 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.182Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2023-07-12T15:30:06.184 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3920141121' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-133619-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:06.185 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]': finished 2023-07-12T15:30:06.185 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-7", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:30:06.185 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: osdmap e93: 8 total, 8 up, 8 in 2023-07-12T15:30:06.185 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:06.185 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:06.185 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3744136116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:06.185 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-7"}]: dispatch 2023-07-12T15:30:06.186 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1579089125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:06.186 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:06.186 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: pgmap v83: 520 pgs: 32 creating+peering, 64 unknown, 424 active+clean; 145 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-07-12T15:30:06.186 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:06.186 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]': finished 2023-07-12T15:30:06.186 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-7"}]': finished 2023-07-12T15:30:06.186 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1579089125' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:06.186 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:06.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:05 smithi145 ceph-mon[95801]: osdmap e94: 8 total, 8 up, 8 in 2023-07-12T15:30:06.438 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3920141121' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-133619-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:06.438 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]': finished 2023-07-12T15:30:06.438 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-7", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:30:06.438 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: osdmap e93: 8 total, 8 up, 8 in 2023-07-12T15:30:06.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:06.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:06.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3744136116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:06.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-7"}]: dispatch 2023-07-12T15:30:06.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1579089125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:06.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:06.439 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: pgmap v83: 520 pgs: 32 creating+peering, 64 unknown, 424 active+clean; 145 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-07-12T15:30:06.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:06.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]': finished 2023-07-12T15:30:06.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-7"}]': finished 2023-07-12T15:30:06.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1579089125' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:06.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:06.440 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[102777]: osdmap e94: 8 total, 8 up, 8 in 2023-07-12T15:30:06.441 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Bus STOPPING 2023-07-12T15:30:06.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3920141121' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi069-133619-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:06.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi069-133932-12"}]': finished 2023-07-12T15:30:06.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-7", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:30:06.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: osdmap e93: 8 total, 8 up, 8 in 2023-07-12T15:30:06.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3160163007' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:06.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]: dispatch 2023-07-12T15:30:06.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3744136116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:06.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-7"}]: dispatch 2023-07-12T15:30:06.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1579089125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:06.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:06.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: pgmap v83: 520 pgs: 32 creating+peering, 64 unknown, 424 active+clean; 145 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-07-12T15:30:06.443 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:06.443 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi069-133932-12"}]': finished 2023-07-12T15:30:06.443 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-7"}]': finished 2023-07-12T15:30:06.443 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1579089125' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:06.443 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-134069-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:06.443 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:05 smithi069 ceph-mon[94241]: osdmap e94: 8 total, 8 up, 8 in 2023-07-12T15:30:06.600 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.183Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2023-07-12T15:30:06.601 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.183Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2023-07-12T15:30:06.601 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.184Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2023-07-12T15:30:06.601 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.184Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.64µs 2023-07-12T15:30:06.601 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.184Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2023-07-12T15:30:06.601 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.193Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2023-07-12T15:30:06.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.194Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2023-07-12T15:30:06.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.194Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.476µs wal_replay_duration=9.695373ms wbl_replay_duration=121ns total_replay_duration=9.730626ms 2023-07-12T15:30:06.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.196Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2023-07-12T15:30:06.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.196Z caller=main.go:1029 level=info msg="TSDB started" 2023-07-12T15:30:06.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.197Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-07-12T15:30:06.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.210Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=13.837207ms db_storage=798ns remote_storage=972ns web_handler=477ns query_engine=652ns scrape=192.795µs scrape_sd=74.826µs notify=14.045µs notify_sd=19.021µs rules=13.296301ms tracing=7.03µs 2023-07-12T15:30:06.603 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.210Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2023-07-12T15:30:06.603 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:30:06.210Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2023-07-12T15:30:06.603 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 bash[124279]: 0d6fd818ca40ae3b9907864bf52c86769f3d9b161462dc76a4357291b97dc739 2023-07-12T15:30:06.603 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:30:06 smithi145 systemd[1]: Started Ceph prometheus.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:30:06.733 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2023-07-12T15:30:06.733 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Bus STOPPED 2023-07-12T15:30:06.734 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Bus STARTING 2023-07-12T15:30:06.734 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Serving on http://:::9283 2023-07-12T15:30:06.734 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Bus STARTED 2023-07-12T15:30:06.734 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Bus STOPPING 2023-07-12T15:30:06.734 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2023-07-12T15:30:06.734 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Bus STOPPED 2023-07-12T15:30:06.734 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Bus STARTING 2023-07-12T15:30:06.735 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Serving on http://:::9283 2023-07-12T15:30:06.735 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Bus STARTED 2023-07-12T15:30:06.735 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Bus STOPPING 2023-07-12T15:30:06.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: Running main() from gmock_main.cc 2023-07-12T15:30:06.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-07-12T15:30:06.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-07-12T15:30:06.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-07-12T15:30:06.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-07-12T15:30:06.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-07-12T15:30:06.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-07-12T15:30:06.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: 2023-07-12T15:30:06.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-07-12T15:30:06.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-07-12T15:30:06.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (8126 ms) 2023-07-12T15:30:06.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-07-12T15:30:06.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3000 ms) 2023-07-12T15:30:06.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-07-12T15:30:06.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2991 ms) 2023-07-12T15:30:06.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-07-12T15:30:06.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2561 ms) 2023-07-12T15:30:06.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-07-12T15:30:06.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3410 ms) 2023-07-12T15:30:06.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-07-12T15:30:06.841 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (2936 ms) 2023-07-12T15:30:06.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-07-12T15:30:06.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (2968 ms) 2023-07-12T15:30:06.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (25992 ms total) 2023-07-12T15:30:06.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: 2023-07-12T15:30:06.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-07-12T15:30:06.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (25992 ms total) 2023-07-12T15:30:06.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-07-12T15:30:06.987 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2023-07-12T15:30:06.988 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Bus STOPPED 2023-07-12T15:30:06.988 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Bus STARTING 2023-07-12T15:30:06.988 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Serving on http://:::9283 2023-07-12T15:30:06.988 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:30:06] ENGINE Bus STARTED 2023-07-12T15:30:07.485 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:30:07.391Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.015181679s 2023-07-12T15:30:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:30:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-12T15:30:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-12T15:30:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2023-07-12T15:30:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2023-07-12T15:30:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-12T15:30:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-12T15:30:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi145.front.sepia.ceph.com:9095"}]: dispatch 2023-07-12T15:30:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi145.front.sepia.ceph.com:9095"}]: dispatch 2023-07-12T15:30:07.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:07.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-12T15:30:07.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-12T15:30:07.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi145.front.sepia.ceph.com:3000"}]: dispatch 2023-07-12T15:30:07.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi145.front.sepia.ceph.com:3000"}]: dispatch 2023-07-12T15:30:07.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:07.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:30:07.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:07.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:30:07.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: osdmap e95: 8 total, 8 up, 8 in 2023-07-12T15:30:07.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-7"}]: dispatch 2023-07-12T15:30:07.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2319569374' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-133619-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:07.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-133619-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:07.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:30:07.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:07.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:07.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-12T15:30:07.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-12T15:30:07.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2023-07-12T15:30:07.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2023-07-12T15:30:07.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:07.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-12T15:30:07.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-12T15:30:07.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi145.front.sepia.ceph.com:9095"}]: dispatch 2023-07-12T15:30:07.991 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi145.front.sepia.ceph.com:9095"}]: dispatch 2023-07-12T15:30:07.992 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:07.992 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-12T15:30:07.992 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-12T15:30:07.992 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi145.front.sepia.ceph.com:3000"}]: dispatch 2023-07-12T15:30:07.992 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi145.front.sepia.ceph.com:3000"}]: dispatch 2023-07-12T15:30:07.992 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:07.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:30:07.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:07.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:30:07.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: osdmap e95: 8 total, 8 up, 8 in 2023-07-12T15:30:07.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-7"}]: dispatch 2023-07-12T15:30:07.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2319569374' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-133619-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:07.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:07 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-133619-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:08.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:30:08.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:08.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:08.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-12T15:30:08.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-07-12T15:30:08.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2023-07-12T15:30:08.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2023-07-12T15:30:08.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:08.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-12T15:30:08.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-07-12T15:30:08.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi145.front.sepia.ceph.com:9095"}]: dispatch 2023-07-12T15:30:08.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi145.front.sepia.ceph.com:9095"}]: dispatch 2023-07-12T15:30:08.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:08.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-12T15:30:08.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-07-12T15:30:08.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi145.front.sepia.ceph.com:3000"}]: dispatch 2023-07-12T15:30:08.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi145.front.sepia.ceph.com:3000"}]: dispatch 2023-07-12T15:30:08.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:08.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:30:08.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:08.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:30:08.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: osdmap e95: 8 total, 8 up, 8 in 2023-07-12T15:30:08.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-7"}]: dispatch 2023-07-12T15:30:08.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2319569374' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-133619-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:08.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:07 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-133619-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:08.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[102777]: pgmap v86: 456 pgs: 32 unknown, 424 active+clean; 145 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2023-07-12T15:30:08.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:08.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-7"}]': finished 2023-07-12T15:30:08.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-133619-8","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:08.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[102777]: osdmap e96: 8 total, 8 up, 8 in 2023-07-12T15:30:08.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1631967941' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:08.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:08.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2101102413' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:08.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:08.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[94241]: pgmap v86: 456 pgs: 32 unknown, 424 active+clean; 145 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2023-07-12T15:30:08.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:08.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-7"}]': finished 2023-07-12T15:30:08.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-133619-8","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:08.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[94241]: osdmap e96: 8 total, 8 up, 8 in 2023-07-12T15:30:08.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1631967941' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:08.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:08.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2101102413' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:08.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:08 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:09.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2023-07-12T15:30:09.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] Global test environment set-up. 2023-07-12T15:30:09.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] 4 tests from LibRadosService 2023-07-12T15:30:09.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-07-12T15:30:09.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5164 ms) 2023-07-12T15:30:09.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-07-12T15:30:09.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.RegisterLate (15 ms) 2023-07-12T15:30:09.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-07-12T15:30:09.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: cluster: 2023-07-12T15:30:09.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: id: dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:30:09.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: health: HEALTH_OK 2023-07-12T15:30:09.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-07-12T15:30:09.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: services: 2023-07-12T15:30:09.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 5m) 2023-07-12T15:30:09.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mgr: y(active, since 93s), standbys: x 2023-07-12T15:30:09.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2023-07-12T15:30:09.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-07-12T15:30:09.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2023-07-12T15:30:09.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-07-12T15:30:09.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: data: 2023-07-12T15:30:09.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pools: 6 pools, 132 pgs 2023-07-12T15:30:09.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: objects: 195 objects, 582 KiB 2023-07-12T15:30:09.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: usage: 218 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:30:09.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pgs: 132 active+clean 2023-07-12T15:30:09.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-07-12T15:30:09.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: io: 2023-07-12T15:30:09.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2023-07-12T15:30:09.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-07-12T15:30:09.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: progress: 2023-07-12T15:30:09.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: Updating node-exporter deployment (+2 -> 2) (7s) 2023-07-12T15:30:09.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [==============..............] (remaining: 7s) 2023-07-12T15:30:09.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-07-12T15:30:09.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-07-12T15:30:09.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: cluster: 2023-07-12T15:30:09.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: id: dc75a490-20c7-11ee-9b31-001a4aab830c 2023-07-12T15:30:09.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: health: HEALTH_OK 2023-07-12T15:30:09.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-07-12T15:30:09.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: services: 2023-07-12T15:30:09.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 5m) 2023-07-12T15:30:09.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mgr: y(active, since 95s), standbys: x 2023-07-12T15:30:09.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2023-07-12T15:30:09.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-07-12T15:30:09.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-07-12T15:30:09.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2023-07-12T15:30:09.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-07-12T15:30:09.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: data: 2023-07-12T15:30:09.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pools: 30 pools, 900 pgs 2023-07-12T15:30:09.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: objects: 195 objects, 582 KiB 2023-07-12T15:30:09.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: usage: 218 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:30:09.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pgs: 55.333% pgs unknown 2023-07-12T15:30:09.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 28.333% pgs not active 2023-07-12T15:30:09.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 498 unknown 2023-07-12T15:30:09.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 255 creating+peering 2023-07-12T15:30:09.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 147 active+clean 2023-07-12T15:30:09.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-07-12T15:30:09.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: io: 2023-07-12T15:30:09.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: client: 899 B/s rd, 0 op/s rd, 0 op/s wr 2023-07-12T15:30:09.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-07-12T15:30:09.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: progress: 2023-07-12T15:30:09.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: Global Recovery Event (0s) 2023-07-12T15:30:09.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [............................] 2023-07-12T15:30:09.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-07-12T15:30:09.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-07-12T15:30:09.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2940 ms) 2023-07-12T15:30:09.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.Status 2023-07-12T15:30:09.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.Status (20144 ms) 2023-07-12T15:30:09.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] 4 tests from LibRadosService (28263 ms total) 2023-07-12T15:30:09.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2023-07-12T15:30:09.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] Global test environment tear-down 2023-07-12T15:30:09.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [==========] 4 tests from 1 test suite ran. (28263 ms total) 2023-07-12T15:30:09.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ PASSED ] 4 tests. 2023-07-12T15:30:09.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:08 smithi145 ceph-mon[95801]: pgmap v86: 456 pgs: 32 unknown, 424 active+clean; 145 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2023-07-12T15:30:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-7"}]': finished 2023-07-12T15:30:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:08 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi069-133619-8","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:08 smithi145 ceph-mon[95801]: osdmap e96: 8 total, 8 up, 8 in 2023-07-12T15:30:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1631967941' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:08 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2101102413' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:09.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:08 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:10.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:10.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:09 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:10.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:09 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:10.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:09 smithi069 ceph-mon[94241]: osdmap e97: 8 total, 8 up, 8 in 2023-07-12T15:30:10.248 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:10.249 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:09 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:10.249 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:09 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:10.249 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:09 smithi069 ceph-mon[102777]: osdmap e97: 8 total, 8 up, 8 in 2023-07-12T15:30:10.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:10.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:09 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi069-133627-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:10.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:09 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:10.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:09 smithi145 ceph-mon[95801]: osdmap e97: 8 total, 8 up, 8 in 2023-07-12T15:30:11.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[102777]: pgmap v89: 456 pgs: 64 unknown, 392 active+clean; 145 MiB data, 749 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:11.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:11.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[102777]: osdmap e98: 8 total, 8 up, 8 in 2023-07-12T15:30:11.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/227032688' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-133619-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:11.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-133619-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:11.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:11.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:11.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[94241]: pgmap v89: 456 pgs: 64 unknown, 392 active+clean; 145 MiB data, 749 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:11.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:11.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[94241]: osdmap e98: 8 total, 8 up, 8 in 2023-07-12T15:30:11.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/227032688' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-133619-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:11.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-133619-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:11.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:11.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:11.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:10 smithi145 ceph-mon[95801]: pgmap v89: 456 pgs: 64 unknown, 392 active+clean; 145 MiB data, 749 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:10 smithi145 ceph-mon[95801]: osdmap e98: 8 total, 8 up, 8 in 2023-07-12T15:30:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/227032688' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-133619-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:10 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-133619-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:11.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:11.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-07-12T15:30:11.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] Global test environment set-up. 2023-07-12T15:30:11.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-07-12T15:30:11.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-07-12T15:30:11.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5042 ms) 2023-07-12T15:30:11.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-07-12T15:30:11.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (44 ms) 2023-07-12T15:30:11.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-07-12T15:30:11.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20080 ms) 2023-07-12T15:30:11.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-07-12T15:30:11.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: attempt 0 of 20 2023-07-12T15:30:11.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5335 ms) 2023-07-12T15:30:11.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30501 ms total) 2023-07-12T15:30:11.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: 2023-07-12T15:30:11.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] Global test environment tear-down 2023-07-12T15:30:11.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30501 ms total) 2023-07-12T15:30:11.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-07-12T15:30:12.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-133619-9","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:12.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-9","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:12.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[94241]: osdmap e99: 8 total, 8 up, 8 in 2023-07-12T15:30:12.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-9", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:30:12.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2449125236' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2023-07-12T15:30:12.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[94241]: pgmap v92: 488 pgs: 64 unknown, 424 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2023-07-12T15:30:12.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[94241]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2023-07-12T15:30:12.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:12.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-133619-9","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:12.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-9","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:12.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[102777]: osdmap e99: 8 total, 8 up, 8 in 2023-07-12T15:30:12.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-9", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:30:12.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2449125236' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2023-07-12T15:30:12.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[102777]: pgmap v92: 488 pgs: 64 unknown, 424 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2023-07-12T15:30:12.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[102777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2023-07-12T15:30:12.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:12.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:11 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi069-133619-9","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:12.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-9","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:12.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:11 smithi145 ceph-mon[95801]: osdmap e99: 8 total, 8 up, 8 in 2023-07-12T15:30:12.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-9", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:30:12.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2449125236' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2023-07-12T15:30:12.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:11 smithi145 ceph-mon[95801]: pgmap v92: 488 pgs: 64 unknown, 424 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2023-07-12T15:30:12.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:11 smithi145 ceph-mon[95801]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2023-07-12T15:30:12.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:13.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4"}]: dispatch 2023-07-12T15:30:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-9", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:30:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:13 smithi145 ceph-mon[95801]: osdmap e100: 8 total, 8 up, 8 in 2023-07-12T15:30:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:13 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4"}]: dispatch 2023-07-12T15:30:13.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-9"}]: dispatch 2023-07-12T15:30:13.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1092963429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-133627-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:13.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:13 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-133627-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:13.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:13.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:13 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:13.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:13 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:13.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4"}]: dispatch 2023-07-12T15:30:13.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-9", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:30:13.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[102777]: osdmap e100: 8 total, 8 up, 8 in 2023-07-12T15:30:13.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4"}]: dispatch 2023-07-12T15:30:13.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-9"}]: dispatch 2023-07-12T15:30:13.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1092963429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-133627-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:13.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-133627-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:13.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:13.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:13.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:13.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4"}]: dispatch 2023-07-12T15:30:13.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-9", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:30:13.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[94241]: osdmap e100: 8 total, 8 up, 8 in 2023-07-12T15:30:13.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4"}]: dispatch 2023-07-12T15:30:13.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-9"}]: dispatch 2023-07-12T15:30:13.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1092963429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-133627-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:13.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-133627-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:13.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:13.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:13.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:13 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:14.836 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:30:15.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4"}]': finished 2023-07-12T15:30:15.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-9"}]': finished 2023-07-12T15:30:15.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-133627-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:15.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: osdmap e101: 8 total, 8 up, 8 in 2023-07-12T15:30:15.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/450441006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6"}]: dispatch 2023-07-12T15:30:15.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3102614237' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-133619-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-9", "mode": "writeback"}]: dispatch 2023-07-12T15:30:15.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6"}]: dispatch 2023-07-12T15:30:15.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-133619-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: pgmap v95: 520 pgs: 128 unknown, 392 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2023-07-12T15:30:15.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:15.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:30:15.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:15.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:15.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:15.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:30:15.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:30:15.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-9", "mode": "writeback"}]': finished 2023-07-12T15:30:15.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-8","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:15.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6"}]': finished 2023-07-12T15:30:15.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-133619-10","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:15.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: osdmap e102: 8 total, 8 up, 8 in 2023-07-12T15:30:15.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-133963-6", "pool2": "test-rados-api-smithi069-133963-6", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-133963-6", "pool2": "test-rados-api-smithi069-133963-6", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:15.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4"}]': finished 2023-07-12T15:30:15.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-9"}]': finished 2023-07-12T15:30:15.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-133627-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:15.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: osdmap e101: 8 total, 8 up, 8 in 2023-07-12T15:30:15.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/450441006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6"}]: dispatch 2023-07-12T15:30:15.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3102614237' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-133619-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-9", "mode": "writeback"}]: dispatch 2023-07-12T15:30:15.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6"}]: dispatch 2023-07-12T15:30:15.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-133619-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: pgmap v95: 520 pgs: 128 unknown, 392 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2023-07-12T15:30:15.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:15.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:30:15.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:15.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:15.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:15.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:30:15.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:30:15.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-9", "mode": "writeback"}]': finished 2023-07-12T15:30:15.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-8","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:15.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6"}]': finished 2023-07-12T15:30:15.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-133619-10","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:15.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: osdmap e102: 8 total, 8 up, 8 in 2023-07-12T15:30:15.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-133963-6", "pool2": "test-rados-api-smithi069-133963-6", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-133963-6", "pool2": "test-rados-api-smithi069-133963-6", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:15.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4"}]': finished 2023-07-12T15:30:15.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-9"}]': finished 2023-07-12T15:30:15.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi069-133627-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:15.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: osdmap e101: 8 total, 8 up, 8 in 2023-07-12T15:30:15.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/450441006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6"}]: dispatch 2023-07-12T15:30:15.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3102614237' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-133619-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-9", "mode": "writeback"}]: dispatch 2023-07-12T15:30:15.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6"}]: dispatch 2023-07-12T15:30:15.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-133619-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: pgmap v95: 520 pgs: 128 unknown, 392 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2023-07-12T15:30:15.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:15.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:30:15.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:15.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:15.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:15.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:30:15.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:30:15.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-9", "mode": "writeback"}]': finished 2023-07-12T15:30:15.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-8","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:15.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi069-133963-4", "tierpool": "test-rados-api-smithi069-133963-6"}]': finished 2023-07-12T15:30:15.248 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi069-133619-10","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:15.249 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: osdmap e102: 8 total, 8 up, 8 in 2023-07-12T15:30:15.249 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2211574976' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-133963-6", "pool2": "test-rados-api-smithi069-133963-6", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.249 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-133963-6", "pool2": "test-rados-api-smithi069-133963-6", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:30:15.249 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:16.080 INFO:tasks.workunit.client.0.smithi069.stdout:watch_notify_pp: flushed 2023-07-12T15:30:16.080 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3021 ms) 2023-07-12T15:30:16.080 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-07-12T15:30:16.080 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: trying... 2023-07-12T15:30:16.080 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94838996518112 notify_id 386547056645 notifier_gid 25124 2023-07-12T15:30:16.080 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: timed out 2023-07-12T15:30:16.081 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2023-07-12T15:30:16.081 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushed 2023-07-12T15:30:16.081 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3015 ms) 2023-07-12T15:30:16.081 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-07-12T15:30:16.081 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: List watches 2023-07-12T15:30:16.082 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 2023-07-12T15:30:16.082 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94838996518112 notify_id 399431958534 notifier_gid 25124 2023-07-12T15:30:16.082 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 done 2023-07-12T15:30:16.082 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: watch_check 2023-07-12T15:30:16.082 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: unwatch2 2023-07-12T15:30:16.082 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2023-07-12T15:30:16.083 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: done 2023-07-12T15:30:16.083 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3110 ms) 2023-07-12T15:30:16.083 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-07-12T15:30:16.083 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: List watches 2023-07-12T15:30:16.083 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 2023-07-12T15:30:16.083 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94838996521264 notify_id 416611827719 notifier_gid 25124 2023-07-12T15:30:16.083 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 done 2023-07-12T15:30:16.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: watch_check 2023-07-12T15:30:16.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: unwatch2 2023-07-12T15:30:16.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2023-07-12T15:30:16.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: done 2023-07-12T15:30:16.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3056 ms) 2023-07-12T15:30:16.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (15968 ms total) 2023-07-12T15:30:16.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: 2023-07-12T15:30:16.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-07-12T15:30:16.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (35287 ms total) 2023-07-12T15:30:16.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-07-12T15:30:16.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: pgmap v96: 520 pgs: 128 unknown, 392 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:16.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 2 stray daemon(s) not managed by cephadm) 2023-07-12T15:30:16.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:30:16.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:16.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:16.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:16.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:16.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-133963-6", "pool2": "test-rados-api-smithi069-133963-6", "yes_i_really_really_mean_it": true}]': finished 2023-07-12T15:30:16.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:30:16.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:16.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: osdmap e103: 8 total, 8 up, 8 in 2023-07-12T15:30:16.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-9"}]: dispatch 2023-07-12T15:30:16.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-133837-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.106 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-133837-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.106 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2197861486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-133627-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:16.106 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-133627-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:16.107 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:16.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: pgmap v96: 520 pgs: 128 unknown, 392 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:16.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 2 stray daemon(s) not managed by cephadm) 2023-07-12T15:30:16.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:30:16.151 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:16.151 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:16.151 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.151 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.151 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.151 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:16.151 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:16.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-133963-6", "pool2": "test-rados-api-smithi069-133963-6", "yes_i_really_really_mean_it": true}]': finished 2023-07-12T15:30:16.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:30:16.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:16.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: osdmap e103: 8 total, 8 up, 8 in 2023-07-12T15:30:16.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-9"}]: dispatch 2023-07-12T15:30:16.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-133837-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-133837-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2197861486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-133627-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:16.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-133627-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:16.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:16.153 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:30:16] "GET /metrics HTTP/1.1" 200 251793 "" "Prometheus/2.43.0" 2023-07-12T15:30:16.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: pgmap v96: 520 pgs: 128 unknown, 392 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:16.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 2 stray daemon(s) not managed by cephadm) 2023-07-12T15:30:16.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:30:16.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:16.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:16.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:16.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:16.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi069-133963-6", "pool2": "test-rados-api-smithi069-133963-6", "yes_i_really_really_mean_it": true}]': finished 2023-07-12T15:30:16.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:30:16.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:16.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: osdmap e103: 8 total, 8 up, 8 in 2023-07-12T15:30:16.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-9"}]: dispatch 2023-07-12T15:30:16.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-133837-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-133837-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:16.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2197861486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-133627-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:16.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-133627-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:16.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:18.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:18 smithi145 ceph-mon[95801]: pgmap v99: 484 pgs: 64 unknown, 420 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:18 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:30:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-9"}]': finished 2023-07-12T15:30:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:18 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-133627-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:18 smithi145 ceph-mon[95801]: osdmap e104: 8 total, 8 up, 8 in 2023-07-12T15:30:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2035024490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-133619-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:18.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:18 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-133837-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]': finished 2023-07-12T15:30:18.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2035024490' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-133619-11","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:18.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:18 smithi145 ceph-mon[95801]: osdmap e105: 8 total, 8 up, 8 in 2023-07-12T15:30:18.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[102777]: pgmap v99: 484 pgs: 64 unknown, 420 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:18.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:30:18.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-9"}]': finished 2023-07-12T15:30:18.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-133627-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:18.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[102777]: osdmap e104: 8 total, 8 up, 8 in 2023-07-12T15:30:18.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2035024490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-133619-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:18.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:18.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-133837-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]': finished 2023-07-12T15:30:18.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2035024490' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-133619-11","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:18.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[102777]: osdmap e105: 8 total, 8 up, 8 in 2023-07-12T15:30:18.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[94241]: pgmap v99: 484 pgs: 64 unknown, 420 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:18.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:30:18.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-9"}]': finished 2023-07-12T15:30:18.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi069-133627-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:18.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[94241]: osdmap e104: 8 total, 8 up, 8 in 2023-07-12T15:30:18.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2035024490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-133619-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:18.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:18.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi069-133837-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]': finished 2023-07-12T15:30:18.494 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2035024490' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi069-133619-11","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:18.494 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:18 smithi069 ceph-mon[94241]: osdmap e105: 8 total, 8 up, 8 in 2023-07-12T15:30:19.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:19 smithi145 ceph-mon[95801]: pgmap v102: 364 pgs: 40 unknown, 324 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:19 smithi145 ceph-mon[95801]: osdmap e106: 8 total, 8 up, 8 in 2023-07-12T15:30:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/489318086' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-133627-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/145536117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:19 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-133627-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:19 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:19.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[94241]: pgmap v102: 364 pgs: 40 unknown, 324 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:19.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[94241]: osdmap e106: 8 total, 8 up, 8 in 2023-07-12T15:30:19.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/489318086' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-133627-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/145536117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-133627-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:19.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:19.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[102777]: pgmap v102: 364 pgs: 40 unknown, 324 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:19.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[102777]: osdmap e106: 8 total, 8 up, 8 in 2023-07-12T15:30:19.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/489318086' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-133627-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/145536117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-133627-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:19.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:20.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:20 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-133627-8","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:20.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:20 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-10","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:20.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-11","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:20.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:20 smithi145 ceph-mon[95801]: osdmap e107: 8 total, 8 up, 8 in 2023-07-12T15:30:20.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3932599778' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-133619-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:20.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:20 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-133619-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:20.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:20.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-133627-8","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:20.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-10","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:20.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-11","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:20.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[94241]: osdmap e107: 8 total, 8 up, 8 in 2023-07-12T15:30:20.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3932599778' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-133619-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:20.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-133619-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:20.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:20.497 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi069-133627-8","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:20.497 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133826-10","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:20.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-11","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:20.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[102777]: osdmap e107: 8 total, 8 up, 8 in 2023-07-12T15:30:20.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3932599778' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-133619-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:20.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-133619-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:20.498 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:21.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: pgmap v105: 460 pgs: 32 creating+peering, 428 active+clean; 146 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 3 op/s 2023-07-12T15:30:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-133619-12","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: osdmap e108: 8 total, 8 up, 8 in 2023-07-12T15:30:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/145536117' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache", "force_nonempty":""}]: dispatch 2023-07-12T15:30:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache", "force_nonempty":""}]: dispatch 2023-07-12T15:30:21.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-11", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:30:21.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:21.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2023-07-12T15:30:21.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2023-07-12T15:30:21.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2023-07-12T15:30:21.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2023-07-12T15:30:21.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2023-07-12T15:30:21.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2023-07-12T15:30:21.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2023-07-12T15:30:21.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2023-07-12T15:30:21.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2023-07-12T15:30:21.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2023-07-12T15:30:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: pgmap v105: 460 pgs: 32 creating+peering, 428 active+clean; 146 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 3 op/s 2023-07-12T15:30:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-133619-12","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: osdmap e108: 8 total, 8 up, 8 in 2023-07-12T15:30:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/145536117' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache", "force_nonempty":""}]: dispatch 2023-07-12T15:30:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache", "force_nonempty":""}]: dispatch 2023-07-12T15:30:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-11", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:30:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2023-07-12T15:30:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2023-07-12T15:30:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2023-07-12T15:30:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2023-07-12T15:30:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2023-07-12T15:30:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2023-07-12T15:30:21.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2023-07-12T15:30:21.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2023-07-12T15:30:21.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2023-07-12T15:30:21.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2023-07-12T15:30:21.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: pgmap v105: 460 pgs: 32 creating+peering, 428 active+clean; 146 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 3 op/s 2023-07-12T15:30:21.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi069-133619-12","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: osdmap e108: 8 total, 8 up, 8 in 2023-07-12T15:30:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/145536117' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache", "force_nonempty":""}]: dispatch 2023-07-12T15:30:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache", "force_nonempty":""}]: dispatch 2023-07-12T15:30:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-11", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:30:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2023-07-12T15:30:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2023-07-12T15:30:21.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2023-07-12T15:30:21.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2023-07-12T15:30:21.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2023-07-12T15:30:21.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2023-07-12T15:30:21.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2023-07-12T15:30:21.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2023-07-12T15:30:21.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2023-07-12T15:30:21.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2023-07-12T15:30:22.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2023-07-12T15:30:22.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2023-07-12T15:30:22.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2023-07-12T15:30:22.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2023-07-12T15:30:22.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2023-07-12T15:30:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2023-07-12T15:30:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2023-07-12T15:30:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2023-07-12T15:30:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2023-07-12T15:30:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2023-07-12T15:30:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: 11.8 deep-scrub starts 2023-07-12T15:30:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: 11.8 deep-scrub ok 2023-07-12T15:30:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache", "force_nonempty":""}]': finished 2023-07-12T15:30:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-11", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:30:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: osdmap e109: 8 total, 8 up, 8 in 2023-07-12T15:30:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-11"}]: dispatch 2023-07-12T15:30:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1401068627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/145536117' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: 11.0 deep-scrub starts 2023-07-12T15:30:22.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: 11.0 deep-scrub ok 2023-07-12T15:30:22.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:22.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-11"}]': finished 2023-07-12T15:30:22.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-9","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:22.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2023-07-12T15:30:22.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2023-07-12T15:30:22.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2023-07-12T15:30:22.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2023-07-12T15:30:22.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2023-07-12T15:30:22.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2023-07-12T15:30:22.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2023-07-12T15:30:22.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2023-07-12T15:30:22.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2023-07-12T15:30:22.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2023-07-12T15:30:22.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: 11.8 deep-scrub starts 2023-07-12T15:30:22.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: 11.8 deep-scrub ok 2023-07-12T15:30:22.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache", "force_nonempty":""}]': finished 2023-07-12T15:30:22.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-11", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:30:22.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: osdmap e109: 8 total, 8 up, 8 in 2023-07-12T15:30:22.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-11"}]: dispatch 2023-07-12T15:30:22.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1401068627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/145536117' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: 11.0 deep-scrub starts 2023-07-12T15:30:22.494 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: 11.0 deep-scrub ok 2023-07-12T15:30:22.494 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:22.494 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-11"}]': finished 2023-07-12T15:30:22.494 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-9","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:22.494 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:22.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: osdmap e110: 8 total, 8 up, 8 in 2023-07-12T15:30:22.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-11", "mode": "writeback"}]: dispatch 2023-07-12T15:30:22.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/145536117' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache"}]: dispatch 2023-07-12T15:30:22.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3201156234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-133619-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-133619-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache"}]: dispatch 2023-07-12T15:30:22.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:22.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: osdmap e110: 8 total, 8 up, 8 in 2023-07-12T15:30:22.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-11", "mode": "writeback"}]: dispatch 2023-07-12T15:30:22.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/145536117' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache"}]: dispatch 2023-07-12T15:30:22.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3201156234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-133619-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.496 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-133619-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.497 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:22 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache"}]: dispatch 2023-07-12T15:30:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2023-07-12T15:30:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2023-07-12T15:30:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2023-07-12T15:30:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2023-07-12T15:30:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2023-07-12T15:30:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2023-07-12T15:30:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2023-07-12T15:30:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2023-07-12T15:30:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2023-07-12T15:30:22.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2023-07-12T15:30:22.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: 11.8 deep-scrub starts 2023-07-12T15:30:22.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: 11.8 deep-scrub ok 2023-07-12T15:30:22.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache", "force_nonempty":""}]': finished 2023-07-12T15:30:22.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-11", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:30:22.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: osdmap e109: 8 total, 8 up, 8 in 2023-07-12T15:30:22.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-11"}]: dispatch 2023-07-12T15:30:22.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1401068627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/145536117' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.605 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: 11.0 deep-scrub starts 2023-07-12T15:30:22.605 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: 11.0 deep-scrub ok 2023-07-12T15:30:22.605 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:22.605 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-11"}]': finished 2023-07-12T15:30:22.605 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-9","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:22.605 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:22.605 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: osdmap e110: 8 total, 8 up, 8 in 2023-07-12T15:30:22.606 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-11", "mode": "writeback"}]: dispatch 2023-07-12T15:30:22.606 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/145536117' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache"}]: dispatch 2023-07-12T15:30:22.606 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3201156234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-133619-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.606 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-133619-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:22.606 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:22 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache"}]: dispatch 2023-07-12T15:30:23.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[94241]: pgmap v109: 492 pgs: 96 unknown, 396 active+clean; 146 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 266 KiB/s wr, 4 op/s 2023-07-12T15:30:23.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[94241]: 11.7 deep-scrub starts 2023-07-12T15:30:23.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[94241]: 11.7 deep-scrub ok 2023-07-12T15:30:23.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[94241]: 11.4 deep-scrub starts 2023-07-12T15:30:23.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[94241]: 11.4 deep-scrub ok 2023-07-12T15:30:23.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:23.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:30:23.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-11", "mode": "writeback"}]': finished 2023-07-12T15:30:23.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-133619-13","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:23.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache"}]': finished 2023-07-12T15:30:23.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[94241]: osdmap e111: 8 total, 8 up, 8 in 2023-07-12T15:30:23.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[102777]: pgmap v109: 492 pgs: 96 unknown, 396 active+clean; 146 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 266 KiB/s wr, 4 op/s 2023-07-12T15:30:23.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[102777]: 11.7 deep-scrub starts 2023-07-12T15:30:23.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[102777]: 11.7 deep-scrub ok 2023-07-12T15:30:23.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[102777]: 11.4 deep-scrub starts 2023-07-12T15:30:23.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[102777]: 11.4 deep-scrub ok 2023-07-12T15:30:23.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:23.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:30:23.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-11", "mode": "writeback"}]': finished 2023-07-12T15:30:23.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-133619-13","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:23.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache"}]': finished 2023-07-12T15:30:23.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:23 smithi069 ceph-mon[102777]: osdmap e111: 8 total, 8 up, 8 in 2023-07-12T15:30:23.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:23 smithi145 ceph-mon[95801]: pgmap v109: 492 pgs: 96 unknown, 396 active+clean; 146 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 266 KiB/s wr, 4 op/s 2023-07-12T15:30:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:23 smithi145 ceph-mon[95801]: 11.7 deep-scrub starts 2023-07-12T15:30:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:23 smithi145 ceph-mon[95801]: 11.7 deep-scrub ok 2023-07-12T15:30:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:23 smithi145 ceph-mon[95801]: 11.4 deep-scrub starts 2023-07-12T15:30:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:23 smithi145 ceph-mon[95801]: 11.4 deep-scrub ok 2023-07-12T15:30:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:23.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:23 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:30:23.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-11", "mode": "writeback"}]': finished 2023-07-12T15:30:23.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:23 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi069-133619-13","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:23.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:23 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133826-10", "tierpool":"test-rados-api-smithi069-133826-10-cache"}]': finished 2023-07-12T15:30:23.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:23 smithi145 ceph-mon[95801]: osdmap e111: 8 total, 8 up, 8 in 2023-07-12T15:30:24.523 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:24 smithi145 ceph-mon[95801]: 11.1 deep-scrub starts 2023-07-12T15:30:24.523 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:24 smithi145 ceph-mon[95801]: 11.1 deep-scrub ok 2023-07-12T15:30:24.524 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:24 smithi145 ceph-mon[95801]: 11.9 deep-scrub starts 2023-07-12T15:30:24.524 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:24 smithi145 ceph-mon[95801]: 11.9 deep-scrub ok 2023-07-12T15:30:24.524 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:24 smithi145 ceph-mon[95801]: 11.5 deep-scrub starts 2023-07-12T15:30:24.524 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:24 smithi145 ceph-mon[95801]: 11.5 deep-scrub ok 2023-07-12T15:30:24.525 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:24.525 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:24 smithi145 ceph-mon[95801]: osdmap e112: 8 total, 8 up, 8 in 2023-07-12T15:30:24.525 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1934576015' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-133627-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:24.526 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:24 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-133627-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:24.526 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:30:24.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[102777]: 11.1 deep-scrub starts 2023-07-12T15:30:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[102777]: 11.1 deep-scrub ok 2023-07-12T15:30:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[102777]: 11.9 deep-scrub starts 2023-07-12T15:30:24.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[102777]: 11.9 deep-scrub ok 2023-07-12T15:30:24.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[102777]: 11.5 deep-scrub starts 2023-07-12T15:30:24.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[102777]: 11.5 deep-scrub ok 2023-07-12T15:30:24.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:24.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[102777]: osdmap e112: 8 total, 8 up, 8 in 2023-07-12T15:30:24.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1934576015' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-133627-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:24.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-133627-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:24.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:30:24.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[94241]: 11.1 deep-scrub starts 2023-07-12T15:30:24.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[94241]: 11.1 deep-scrub ok 2023-07-12T15:30:24.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[94241]: 11.9 deep-scrub starts 2023-07-12T15:30:24.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[94241]: 11.9 deep-scrub ok 2023-07-12T15:30:24.742 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[94241]: 11.5 deep-scrub starts 2023-07-12T15:30:24.742 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[94241]: 11.5 deep-scrub ok 2023-07-12T15:30:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[94241]: osdmap e112: 8 total, 8 up, 8 in 2023-07-12T15:30:24.743 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1934576015' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-133627-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:24.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-133627-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:24.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:30:24.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:30:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:30:24.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: Running main() from gmock_main.cc 2023-07-12T15:30:24.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-07-12T15:30:24.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] Global test environment set-up. 2023-07-12T15:30:24.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-07-12T15:30:24.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-07-12T15:30:24.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolList (8229 ms) 2023-07-12T15:30:24.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-07-12T15:30:24.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (2996 ms) 2023-07-12T15:30:24.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-07-12T15:30:24.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3036 ms) 2023-07-12T15:30:24.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-07-12T15:30:24.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2531 ms) 2023-07-12T15:30:24.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-07-12T15:30:24.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3394 ms) 2023-07-12T15:30:24.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-07-12T15:30:24.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5904 ms) 2023-07-12T15:30:24.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-07-12T15:30:24.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5181 ms) 2023-07-12T15:30:24.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-07-12T15:30:24.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4838 ms) 2023-07-12T15:30:24.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-07-12T15:30:24.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8052 ms) 2023-07-12T15:30:24.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] 9 tests from LibRadosPools (44161 ms total) 2023-07-12T15:30:24.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: 2023-07-12T15:30:24.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] Global test environment tear-down 2023-07-12T15:30:24.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (44161 ms total) 2023-07-12T15:30:24.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ PASSED ] 9 tests. 2023-07-12T15:30:25.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:25 smithi145 ceph-mon[95801]: 11.3 deep-scrub starts 2023-07-12T15:30:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:25 smithi145 ceph-mon[95801]: 11.3 deep-scrub ok 2023-07-12T15:30:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:25 smithi145 ceph-mon[95801]: pgmap v112: 428 pgs: 32 unknown, 396 active+clean; 146 MiB data, 768 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:25 smithi145 ceph-mon[95801]: 11.2 deep-scrub starts 2023-07-12T15:30:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:25 smithi145 ceph-mon[95801]: 11.2 deep-scrub ok 2023-07-12T15:30:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:25 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-133627-10","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:30:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:25 smithi145 ceph-mon[95801]: osdmap e113: 8 total, 8 up, 8 in 2023-07-12T15:30:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-11"}]: dispatch 2023-07-12T15:30:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3558416910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-133619-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:25.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-133619-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:25.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[94241]: 11.3 deep-scrub starts 2023-07-12T15:30:25.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[94241]: 11.3 deep-scrub ok 2023-07-12T15:30:25.745 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[94241]: pgmap v112: 428 pgs: 32 unknown, 396 active+clean; 146 MiB data, 768 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[94241]: 11.2 deep-scrub starts 2023-07-12T15:30:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[94241]: 11.2 deep-scrub ok 2023-07-12T15:30:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-133627-10","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:30:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[94241]: osdmap e113: 8 total, 8 up, 8 in 2023-07-12T15:30:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-11"}]: dispatch 2023-07-12T15:30:25.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3558416910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-133619-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:25.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-133619-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:25.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[102777]: 11.3 deep-scrub starts 2023-07-12T15:30:25.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[102777]: 11.3 deep-scrub ok 2023-07-12T15:30:25.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[102777]: pgmap v112: 428 pgs: 32 unknown, 396 active+clean; 146 MiB data, 768 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:25.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[102777]: 11.2 deep-scrub starts 2023-07-12T15:30:25.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[102777]: 11.2 deep-scrub ok 2023-07-12T15:30:25.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:25.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:25.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi069-133627-10","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:25.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:30:25.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[102777]: osdmap e113: 8 total, 8 up, 8 in 2023-07-12T15:30:25.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-11"}]: dispatch 2023-07-12T15:30:25.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3558416910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-133619-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:25.749 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-133619-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:26.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:30:26] "GET /metrics HTTP/1.1" 200 251793 "" "Prometheus/2.43.0" 2023-07-12T15:30:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:26 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:30:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-11"}]': finished 2023-07-12T15:30:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-133619-14","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:26 smithi069 ceph-mon[102777]: osdmap e114: 8 total, 8 up, 8 in 2023-07-12T15:30:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:26 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:30:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-11"}]': finished 2023-07-12T15:30:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-133619-14","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:26 smithi069 ceph-mon[94241]: osdmap e114: 8 total, 8 up, 8 in 2023-07-12T15:30:26.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:26 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:30:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-11"}]': finished 2023-07-12T15:30:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi069-133619-14","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:26 smithi145 ceph-mon[95801]: osdmap e114: 8 total, 8 up, 8 in 2023-07-12T15:30:26.898 INFO:tasks.workunit.client.0.smithi069.stdout:sc: started 0x55a288e8ed10 2023-07-12T15:30:26.899 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 2 aios 2023-07-12T15:30:26.899 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55a288b55610 2023-07-12T15:30:26.899 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x55a288e8ed10 2023-07-12T15:30:26.899 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: done. 2023-07-12T15:30:26.899 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (10974 ms) 2023-07-12T15:30:26.899 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (10974 ms total) 2023-07-12T15:30:26.900 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:30:26.900 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-07-12T15:30:26.900 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-07-12T15:30:26.900 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-07-12T15:30:26.900 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-07-12T15:30:26.900 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.Exec (6 ms) 2023-07-12T15:30:26.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-07-12T15:30:26.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (3 ms) 2023-07-12T15:30:26.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-07-12T15:30:26.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3 ms) 2023-07-12T15:30:26.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-07-12T15:30:26.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4288 ms) 2023-07-12T15:30:26.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-07-12T15:30:26.901 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-07-12T15:30:26.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-07-12T15:30:26.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-07-12T15:30:26.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-07-12T15:30:26.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (27851 ms) 2023-07-12T15:30:26.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 8 tests from LibRadosMisc (32151 ms total) 2023-07-12T15:30:26.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2023-07-12T15:30:26.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] Global test environment tear-down 2023-07-12T15:30:26.902 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [==========] 11 tests from 4 test suites ran. (46168 ms total) 2023-07-12T15:30:26.903 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ PASSED ] 11 tests. 2023-07-12T15:30:27.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:27 smithi069 ceph-mon[102777]: pgmap v115: 396 pgs: 32 unknown, 364 active+clean; 217 MiB data, 978 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 9 op/s 2023-07-12T15:30:27.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:27.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:27 smithi069 ceph-mon[102777]: osdmap e115: 8 total, 8 up, 8 in 2023-07-12T15:30:27.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2846460217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-133627-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:27.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:27.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:27 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:27.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:27 smithi069 ceph-mon[94241]: pgmap v115: 396 pgs: 32 unknown, 364 active+clean; 217 MiB data, 978 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 9 op/s 2023-07-12T15:30:27.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:27.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:27 smithi069 ceph-mon[94241]: osdmap e115: 8 total, 8 up, 8 in 2023-07-12T15:30:27.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2846460217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-133627-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:27.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:27.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:27 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:27.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:27 smithi145 ceph-mon[95801]: pgmap v115: 396 pgs: 32 unknown, 364 active+clean; 217 MiB data, 978 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 9 op/s 2023-07-12T15:30:27.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:27.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:27 smithi145 ceph-mon[95801]: osdmap e115: 8 total, 8 up, 8 in 2023-07-12T15:30:27.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2846460217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-133627-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:27.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:27.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:27 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:28.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[102777]: 11.6 deep-scrub starts 2023-07-12T15:30:28.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[102777]: 11.6 deep-scrub ok 2023-07-12T15:30:28.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:28.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2846460217' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-133627-11","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:28.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]': finished 2023-07-12T15:30:28.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[102777]: osdmap e116: 8 total, 8 up, 8 in 2023-07-12T15:30:28.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:28.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1700375692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi069-133619-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:28.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi069-133619-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:28.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:28.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:28.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[94241]: 11.6 deep-scrub starts 2023-07-12T15:30:28.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[94241]: 11.6 deep-scrub ok 2023-07-12T15:30:28.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:28.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2846460217' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-133627-11","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:28.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]': finished 2023-07-12T15:30:28.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[94241]: osdmap e116: 8 total, 8 up, 8 in 2023-07-12T15:30:28.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:28.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1700375692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi069-133619-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:28.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi069-133619-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:28.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:28.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:28 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:28.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:28 smithi145 ceph-mon[95801]: 11.6 deep-scrub starts 2023-07-12T15:30:28.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:28 smithi145 ceph-mon[95801]: 11.6 deep-scrub ok 2023-07-12T15:30:28.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:28.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2846460217' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi069-133627-11","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:28.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:28 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi069-133837-10"}]': finished 2023-07-12T15:30:28.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:28 smithi145 ceph-mon[95801]: osdmap e116: 8 total, 8 up, 8 in 2023-07-12T15:30:28.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:28.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1700375692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi069-133619-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:28.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:28 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi069-133619-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:28.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1291866382' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:28.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:28 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]: dispatch 2023-07-12T15:30:29.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[102777]: pgmap v118: 388 pgs: 96 unknown, 292 active+clean; 217 MiB data, 978 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2023-07-12T15:30:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:30:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[102777]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-13","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi069-133619-15","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]': finished 2023-07-12T15:30:29.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[102777]: osdmap e117: 8 total, 8 up, 8 in 2023-07-12T15:30:29.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:29.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:29.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[94241]: pgmap v118: 388 pgs: 96 unknown, 292 active+clean; 217 MiB data, 978 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2023-07-12T15:30:29.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:29.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:30:29.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:29.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[94241]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:29.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-13","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:29.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi069-133619-15","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:29.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]': finished 2023-07-12T15:30:29.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[94241]: osdmap e117: 8 total, 8 up, 8 in 2023-07-12T15:30:29.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:29.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:29.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:29.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:29 smithi145 ceph-mon[95801]: pgmap v118: 388 pgs: 96 unknown, 292 active+clean; 217 MiB data, 978 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2023-07-12T15:30:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:29 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:30:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:29 smithi145 ceph-mon[95801]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-13","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:29.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:29 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi069-133619-15","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:29.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:29 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi069-133837-10"}]': finished 2023-07-12T15:30:29.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:29 smithi145 ceph-mon[95801]: osdmap e117: 8 total, 8 up, 8 in 2023-07-12T15:30:29.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:29.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:29.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:30.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:30.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:30.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:30 smithi069 ceph-mon[102777]: osdmap e118: 8 total, 8 up, 8 in 2023-07-12T15:30:30.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/231215213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-133627-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:30.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-133837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:30.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:30.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:30.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:30 smithi069 ceph-mon[94241]: osdmap e118: 8 total, 8 up, 8 in 2023-07-12T15:30:30.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/231215213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-133627-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:30.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-133837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:30.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:30.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:30.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:30 smithi145 ceph-mon[95801]: osdmap e118: 8 total, 8 up, 8 in 2023-07-12T15:30:30.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/231215213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-133627-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:30.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-133837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:31 smithi069 ceph-mon[102777]: pgmap v121: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s wr, 21 op/s 2023-07-12T15:30:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/231215213' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-133627-12","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:31 smithi069 ceph-mon[102777]: osdmap e119: 8 total, 8 up, 8 in 2023-07-12T15:30:31.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1414393821' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-133619-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:31.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:31 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-133619-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:31.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:31 smithi069 ceph-mon[94241]: pgmap v121: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s wr, 21 op/s 2023-07-12T15:30:31.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:31.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/231215213' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-133627-12","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:31.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:31 smithi069 ceph-mon[94241]: osdmap e119: 8 total, 8 up, 8 in 2023-07-12T15:30:31.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1414393821' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-133619-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:31.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:31 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-133619-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:31.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:31 smithi145 ceph-mon[95801]: pgmap v121: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s wr, 21 op/s 2023-07-12T15:30:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/231215213' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi069-133627-12","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:31 smithi145 ceph-mon[95801]: osdmap e119: 8 total, 8 up, 8 in 2023-07-12T15:30:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1414393821' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-133619-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:31 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-133619-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:32.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-133837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]': finished 2023-07-12T15:30:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:32 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-133619-16","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:32 smithi069 ceph-mon[102777]: osdmap e120: 8 total, 8 up, 8 in 2023-07-12T15:30:32.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:32.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-133837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]': finished 2023-07-12T15:30:32.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:32 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-133619-16","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:32.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:32 smithi069 ceph-mon[94241]: osdmap e120: 8 total, 8 up, 8 in 2023-07-12T15:30:32.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:32.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi069-133837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]': finished 2023-07-12T15:30:32.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:32 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi069-133619-16","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:32.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:32 smithi145 ceph-mon[95801]: osdmap e120: 8 total, 8 up, 8 in 2023-07-12T15:30:33.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:33 smithi069 ceph-mon[102777]: pgmap v124: 364 pgs: 40 unknown, 324 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 25 KiB/s wr, 26 op/s 2023-07-12T15:30:33.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:33.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:33 smithi069 ceph-mon[102777]: osdmap e121: 8 total, 8 up, 8 in 2023-07-12T15:30:33.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1536662506' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-133627-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:33.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:33 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-133627-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:33.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-13", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:30:33.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:33 smithi069 ceph-mon[94241]: pgmap v124: 364 pgs: 40 unknown, 324 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 25 KiB/s wr, 26 op/s 2023-07-12T15:30:33.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:33.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:33 smithi069 ceph-mon[94241]: osdmap e121: 8 total, 8 up, 8 in 2023-07-12T15:30:33.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1536662506' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-133627-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:33.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:33 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-133627-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:33.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-13", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:30:33.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:33 smithi145 ceph-mon[95801]: pgmap v124: 364 pgs: 40 unknown, 324 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 25 KiB/s wr, 26 op/s 2023-07-12T15:30:33.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:33.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:33 smithi145 ceph-mon[95801]: osdmap e121: 8 total, 8 up, 8 in 2023-07-12T15:30:33.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1536662506' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-133627-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:33.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:33 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-133627-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:33.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-13", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:30:34.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:34.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-133627-13","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:34.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-13", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:30:34.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[102777]: osdmap e122: 8 total, 8 up, 8 in 2023-07-12T15:30:34.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-13"}]: dispatch 2023-07-12T15:30:34.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2093108249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-133619-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:34.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-133619-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:34.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:34.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-133627-13","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:34.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-13", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:30:34.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[94241]: osdmap e122: 8 total, 8 up, 8 in 2023-07-12T15:30:34.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-13"}]: dispatch 2023-07-12T15:30:34.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2093108249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-133619-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:34.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:34 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-133619-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:34.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:30:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:30:34.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:34.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:34 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi069-133627-13","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:34.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-13", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:30:34.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:34 smithi145 ceph-mon[95801]: osdmap e122: 8 total, 8 up, 8 in 2023-07-12T15:30:34.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-13"}]: dispatch 2023-07-12T15:30:34.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2093108249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-133619-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:34.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:34 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-133619-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:35.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[94241]: pgmap v127: 396 pgs: 72 unknown, 324 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:35.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:35.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:35.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-13"}]': finished 2023-07-12T15:30:35.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-133619-17","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:35.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[94241]: osdmap e123: 8 total, 8 up, 8 in 2023-07-12T15:30:35.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-13", "mode": "writeback"}]: dispatch 2023-07-12T15:30:35.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[102777]: pgmap v127: 396 pgs: 72 unknown, 324 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:35.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:35.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:35.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-13"}]': finished 2023-07-12T15:30:35.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-133619-17","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:35.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[102777]: osdmap e123: 8 total, 8 up, 8 in 2023-07-12T15:30:35.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-13", "mode": "writeback"}]: dispatch 2023-07-12T15:30:35.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:35 smithi145 ceph-mon[95801]: pgmap v127: 396 pgs: 72 unknown, 324 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:35.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:35 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:35.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:35.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-13"}]': finished 2023-07-12T15:30:35.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:35 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi069-133619-17","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:35.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:35 smithi145 ceph-mon[95801]: osdmap e123: 8 total, 8 up, 8 in 2023-07-12T15:30:35.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-13", "mode": "writeback"}]: dispatch 2023-07-12T15:30:36.461 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:30:36] "GET /metrics HTTP/1.1" 200 239935 "" "Prometheus/2.43.0" 2023-07-12T15:30:36.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:36.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:30:36.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-13", "mode": "writeback"}]': finished 2023-07-12T15:30:36.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: osdmap e124: 8 total, 8 up, 8 in 2023-07-12T15:30:36.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1952525742' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-133627-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:36.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-133627-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:36.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.0"}]: dispatch 2023-07-12T15:30:36.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.1"}]: dispatch 2023-07-12T15:30:36.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.2"}]: dispatch 2023-07-12T15:30:36.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.3"}]: dispatch 2023-07-12T15:30:36.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.4"}]: dispatch 2023-07-12T15:30:36.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.5"}]: dispatch 2023-07-12T15:30:36.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.6"}]: dispatch 2023-07-12T15:30:36.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.7"}]: dispatch 2023-07-12T15:30:36.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.8"}]: dispatch 2023-07-12T15:30:36.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.9"}]: dispatch 2023-07-12T15:30:36.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:36.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:30:36.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-13", "mode": "writeback"}]': finished 2023-07-12T15:30:36.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: osdmap e124: 8 total, 8 up, 8 in 2023-07-12T15:30:36.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1952525742' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-133627-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:36.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-133627-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:36.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.0"}]: dispatch 2023-07-12T15:30:36.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.1"}]: dispatch 2023-07-12T15:30:36.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.2"}]: dispatch 2023-07-12T15:30:36.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.3"}]: dispatch 2023-07-12T15:30:36.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.4"}]: dispatch 2023-07-12T15:30:36.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.5"}]: dispatch 2023-07-12T15:30:36.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.6"}]: dispatch 2023-07-12T15:30:36.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.7"}]: dispatch 2023-07-12T15:30:36.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.8"}]: dispatch 2023-07-12T15:30:36.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.9"}]: dispatch 2023-07-12T15:30:36.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:36.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:30:36.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-13", "mode": "writeback"}]': finished 2023-07-12T15:30:36.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: osdmap e124: 8 total, 8 up, 8 in 2023-07-12T15:30:36.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1952525742' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-133627-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:36.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-133627-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:36.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.0"}]: dispatch 2023-07-12T15:30:36.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.1"}]: dispatch 2023-07-12T15:30:36.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.2"}]: dispatch 2023-07-12T15:30:36.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.3"}]: dispatch 2023-07-12T15:30:36.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.4"}]: dispatch 2023-07-12T15:30:36.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.5"}]: dispatch 2023-07-12T15:30:36.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.6"}]: dispatch 2023-07-12T15:30:36.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.7"}]: dispatch 2023-07-12T15:30:36.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.8"}]: dispatch 2023-07-12T15:30:36.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "96.9"}]: dispatch 2023-07-12T15:30:37.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: pgmap v130: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2023-07-12T15:30:37.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.0"}]: dispatch 2023-07-12T15:30:37.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.1"}]: dispatch 2023-07-12T15:30:37.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.2"}]: dispatch 2023-07-12T15:30:37.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.3"}]: dispatch 2023-07-12T15:30:37.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.4"}]: dispatch 2023-07-12T15:30:37.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.5"}]: dispatch 2023-07-12T15:30:37.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.6"}]: dispatch 2023-07-12T15:30:37.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.7"}]: dispatch 2023-07-12T15:30:37.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.8"}]: dispatch 2023-07-12T15:30:37.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.9"}]: dispatch 2023-07-12T15:30:37.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:37.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:37.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-133627-14","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:37.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: osdmap e125: 8 total, 8 up, 8 in 2023-07-12T15:30:37.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1416771333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-133619-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:37.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: pgmap v130: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2023-07-12T15:30:37.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.0"}]: dispatch 2023-07-12T15:30:37.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.1"}]: dispatch 2023-07-12T15:30:37.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.2"}]: dispatch 2023-07-12T15:30:37.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.3"}]: dispatch 2023-07-12T15:30:37.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.4"}]: dispatch 2023-07-12T15:30:37.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.5"}]: dispatch 2023-07-12T15:30:37.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.6"}]: dispatch 2023-07-12T15:30:37.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.7"}]: dispatch 2023-07-12T15:30:37.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.8"}]: dispatch 2023-07-12T15:30:37.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.9"}]: dispatch 2023-07-12T15:30:37.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:37.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:37.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-133627-14","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:37.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: osdmap e125: 8 total, 8 up, 8 in 2023-07-12T15:30:37.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1416771333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-133619-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:37.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: pgmap v130: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2023-07-12T15:30:37.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.0"}]: dispatch 2023-07-12T15:30:37.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.1"}]: dispatch 2023-07-12T15:30:37.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.2"}]: dispatch 2023-07-12T15:30:37.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.3"}]: dispatch 2023-07-12T15:30:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.4"}]: dispatch 2023-07-12T15:30:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.5"}]: dispatch 2023-07-12T15:30:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.6"}]: dispatch 2023-07-12T15:30:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.7"}]: dispatch 2023-07-12T15:30:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.8"}]: dispatch 2023-07-12T15:30:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "96.9"}]: dispatch 2023-07-12T15:30:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:37.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:37.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi069-133627-14","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:37.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: osdmap e125: 8 total, 8 up, 8 in 2023-07-12T15:30:37.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1416771333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-133619-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:38.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[94241]: 96.7 deep-scrub starts 2023-07-12T15:30:38.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[94241]: 96.7 deep-scrub ok 2023-07-12T15:30:38.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[94241]: 96.4 deep-scrub starts 2023-07-12T15:30:38.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[94241]: 96.4 deep-scrub ok 2023-07-12T15:30:38.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[94241]: 96.8 scrub starts 2023-07-12T15:30:38.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[94241]: 96.8 scrub ok 2023-07-12T15:30:38.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:38.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1416771333' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-133619-18","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:38.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[94241]: osdmap e126: 8 total, 8 up, 8 in 2023-07-12T15:30:38.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[94241]: 96.6 deep-scrub starts 2023-07-12T15:30:38.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[94241]: 96.6 deep-scrub ok 2023-07-12T15:30:38.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[102777]: 96.7 deep-scrub starts 2023-07-12T15:30:38.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[102777]: 96.7 deep-scrub ok 2023-07-12T15:30:38.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[102777]: 96.4 deep-scrub starts 2023-07-12T15:30:38.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[102777]: 96.4 deep-scrub ok 2023-07-12T15:30:38.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[102777]: 96.8 scrub starts 2023-07-12T15:30:38.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[102777]: 96.8 scrub ok 2023-07-12T15:30:38.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:38.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1416771333' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-133619-18","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:38.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[102777]: osdmap e126: 8 total, 8 up, 8 in 2023-07-12T15:30:38.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[102777]: 96.6 deep-scrub starts 2023-07-12T15:30:38.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:38 smithi069 ceph-mon[102777]: 96.6 deep-scrub ok 2023-07-12T15:30:38.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:38 smithi145 ceph-mon[95801]: 96.7 deep-scrub starts 2023-07-12T15:30:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:38 smithi145 ceph-mon[95801]: 96.7 deep-scrub ok 2023-07-12T15:30:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:38 smithi145 ceph-mon[95801]: 96.4 deep-scrub starts 2023-07-12T15:30:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:38 smithi145 ceph-mon[95801]: 96.4 deep-scrub ok 2023-07-12T15:30:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:38 smithi145 ceph-mon[95801]: 96.8 scrub starts 2023-07-12T15:30:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:38 smithi145 ceph-mon[95801]: 96.8 scrub ok 2023-07-12T15:30:38.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:38.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1416771333' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi069-133619-18","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:38.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:38 smithi145 ceph-mon[95801]: osdmap e126: 8 total, 8 up, 8 in 2023-07-12T15:30:38.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:38 smithi145 ceph-mon[95801]: 96.6 deep-scrub starts 2023-07-12T15:30:38.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:38 smithi145 ceph-mon[95801]: 96.6 deep-scrub ok 2023-07-12T15:30:39.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[94241]: pgmap v133: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2023-07-12T15:30:39.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[94241]: 96.9 scrub starts 2023-07-12T15:30:39.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[94241]: 96.9 scrub ok 2023-07-12T15:30:39.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[94241]: 96.3 scrub starts 2023-07-12T15:30:39.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[94241]: 96.3 scrub ok 2023-07-12T15:30:39.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:39.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[94241]: osdmap e127: 8 total, 8 up, 8 in 2023-07-12T15:30:39.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3623969410' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-133627-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:39.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-133627-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:39.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[102777]: pgmap v133: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2023-07-12T15:30:39.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[102777]: 96.9 scrub starts 2023-07-12T15:30:39.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[102777]: 96.9 scrub ok 2023-07-12T15:30:39.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[102777]: 96.3 scrub starts 2023-07-12T15:30:39.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[102777]: 96.3 scrub ok 2023-07-12T15:30:39.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:39.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[102777]: osdmap e127: 8 total, 8 up, 8 in 2023-07-12T15:30:39.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3623969410' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-133627-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:39.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:39 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-133627-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:39.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:39 smithi145 ceph-mon[95801]: pgmap v133: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2023-07-12T15:30:39.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:39 smithi145 ceph-mon[95801]: 96.9 scrub starts 2023-07-12T15:30:39.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:39 smithi145 ceph-mon[95801]: 96.9 scrub ok 2023-07-12T15:30:39.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:39 smithi145 ceph-mon[95801]: 96.3 scrub starts 2023-07-12T15:30:39.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:39 smithi145 ceph-mon[95801]: 96.3 scrub ok 2023-07-12T15:30:39.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:39.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:39 smithi145 ceph-mon[95801]: osdmap e127: 8 total, 8 up, 8 in 2023-07-12T15:30:39.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3623969410' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-133627-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:39.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:39 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-133627-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:40.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:40 smithi069 ceph-mon[102777]: 96.5 scrub starts 2023-07-12T15:30:40.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:40 smithi069 ceph-mon[102777]: 96.5 scrub ok 2023-07-12T15:30:40.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:40.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:40 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-133627-15","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:40.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:40 smithi069 ceph-mon[102777]: osdmap e128: 8 total, 8 up, 8 in 2023-07-12T15:30:40.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2152328689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-133619-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:40.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:40 smithi069 ceph-mon[94241]: 96.5 scrub starts 2023-07-12T15:30:40.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:40 smithi069 ceph-mon[94241]: 96.5 scrub ok 2023-07-12T15:30:40.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:40.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:40 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-133627-15","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:40.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:40 smithi069 ceph-mon[94241]: osdmap e128: 8 total, 8 up, 8 in 2023-07-12T15:30:40.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2152328689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-133619-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:40.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:40 smithi145 ceph-mon[95801]: 96.5 scrub starts 2023-07-12T15:30:40.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:40 smithi145 ceph-mon[95801]: 96.5 scrub ok 2023-07-12T15:30:40.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:40.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:40 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi069-133627-15","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:40.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:40 smithi145 ceph-mon[95801]: osdmap e128: 8 total, 8 up, 8 in 2023-07-12T15:30:40.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2152328689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-133619-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:41.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[102777]: pgmap v136: 396 pgs: 32 unknown, 32 creating+peering, 332 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.0 KiB/s wr, 77 op/s 2023-07-12T15:30:41.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:41.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2152328689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-133619-19","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:41.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[102777]: osdmap e129: 8 total, 8 up, 8 in 2023-07-12T15:30:41.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:41.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[102777]: 96.1 scrub starts 2023-07-12T15:30:41.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[102777]: 96.1 scrub ok 2023-07-12T15:30:41.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[94241]: pgmap v136: 396 pgs: 32 unknown, 32 creating+peering, 332 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.0 KiB/s wr, 77 op/s 2023-07-12T15:30:41.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:41.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2152328689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-133619-19","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:41.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[94241]: osdmap e129: 8 total, 8 up, 8 in 2023-07-12T15:30:41.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:41.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[94241]: 96.1 scrub starts 2023-07-12T15:30:41.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:41 smithi069 ceph-mon[94241]: 96.1 scrub ok 2023-07-12T15:30:41.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:41 smithi145 ceph-mon[95801]: pgmap v136: 396 pgs: 32 unknown, 32 creating+peering, 332 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.0 KiB/s wr, 77 op/s 2023-07-12T15:30:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2152328689' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi069-133619-19","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:41 smithi145 ceph-mon[95801]: osdmap e129: 8 total, 8 up, 8 in 2023-07-12T15:30:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:41 smithi145 ceph-mon[95801]: 96.1 scrub starts 2023-07-12T15:30:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:41 smithi145 ceph-mon[95801]: 96.1 scrub ok 2023-07-12T15:30:42.816 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: Running main() from gmock_main.cc 2023-07-12T15:30:42.817 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-07-12T15:30:42.817 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] Global test environment set-up. 2023-07-12T15:30:42.817 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-07-12T15:30:42.817 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-07-12T15:30:42.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2856 ms) 2023-07-12T15:30:42.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-07-12T15:30:42.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (1542 ms) 2023-07-12T15:30:42.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-07-12T15:30:42.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2270 ms) 2023-07-12T15:30:42.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-07-12T15:30:42.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1743 ms) 2023-07-12T15:30:42.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8411 ms total) 2023-07-12T15:30:42.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2023-07-12T15:30:42.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-07-12T15:30:42.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-07-12T15:30:42.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4340 ms) 2023-07-12T15:30:42.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-07-12T15:30:42.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (3970 ms) 2023-07-12T15:30:42.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-07-12T15:30:42.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (6228 ms) 2023-07-12T15:30:42.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (14538 ms total) 2023-07-12T15:30:42.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2023-07-12T15:30:42.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-07-12T15:30:42.821 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-07-12T15:30:42.821 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2896 ms) 2023-07-12T15:30:42.821 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-07-12T15:30:42.821 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1621 ms) 2023-07-12T15:30:42.821 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-07-12T15:30:42.821 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2085 ms) 2023-07-12T15:30:42.822 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-07-12T15:30:42.822 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2033 ms) 2023-07-12T15:30:42.822 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8636 ms total) 2023-07-12T15:30:42.822 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2023-07-12T15:30:42.822 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-07-12T15:30:42.822 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-07-12T15:30:42.822 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4029 ms) 2023-07-12T15:30:42.823 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-07-12T15:30:42.823 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4019 ms) 2023-07-12T15:30:42.823 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8048 ms total) 2023-07-12T15:30:42.823 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2023-07-12T15:30:42.823 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] Global test environment tear-down 2023-07-12T15:30:42.823 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (62072 ms total) 2023-07-12T15:30:42.823 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-07-12T15:30:42.824 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:30:42.824 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134067 2023-07-12T15:30:42.824 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 134067 2023-07-12T15:30:42.824 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:30:42.825 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133646 2023-07-12T15:30:42.825 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133646 2023-07-12T15:30:42.825 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:30:42.825 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133939 2023-07-12T15:30:42.825 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133939 2023-07-12T15:30:42.825 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:30:42.825 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133844 2023-07-12T15:30:42.826 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133844 2023-07-12T15:30:43.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]': finished 2023-07-12T15:30:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:42 smithi145 ceph-mon[95801]: osdmap e130: 8 total, 8 up, 8 in 2023-07-12T15:30:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2162734764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi069-133627-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:42 smithi145 ceph-mon[95801]: pgmap v139: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2023-07-12T15:30:43.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]': finished 2023-07-12T15:30:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:42 smithi069 ceph-mon[102777]: osdmap e130: 8 total, 8 up, 8 in 2023-07-12T15:30:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2162734764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi069-133627-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:42 smithi069 ceph-mon[102777]: pgmap v139: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2023-07-12T15:30:43.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:43.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]': finished 2023-07-12T15:30:43.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:42 smithi069 ceph-mon[94241]: osdmap e130: 8 total, 8 up, 8 in 2023-07-12T15:30:43.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]: dispatch 2023-07-12T15:30:43.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2162734764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi069-133627-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:43.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:42 smithi069 ceph-mon[94241]: pgmap v139: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2023-07-12T15:30:44.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: 96.0 scrub starts 2023-07-12T15:30:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: 96.0 scrub ok 2023-07-12T15:30:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]': finished 2023-07-12T15:30:44.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2162734764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi069-133627-16","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:44.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: osdmap e131: 8 total, 8 up, 8 in 2023-07-12T15:30:44.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/97698434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-133619-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:44.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:44.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:44.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:44.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:43 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:30:44.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: 96.0 scrub starts 2023-07-12T15:30:44.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: 96.0 scrub ok 2023-07-12T15:30:44.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:44.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]': finished 2023-07-12T15:30:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2162734764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi069-133627-16","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: osdmap e131: 8 total, 8 up, 8 in 2023-07-12T15:30:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/97698434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-133619-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:44.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:44.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:30:44.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: 96.0 scrub starts 2023-07-12T15:30:44.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: 96.0 scrub ok 2023-07-12T15:30:44.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:44.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:44.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3293335457' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi069-133837-15"}]': finished 2023-07-12T15:30:44.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2162734764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi069-133627-16","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:44.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: osdmap e131: 8 total, 8 up, 8 in 2023-07-12T15:30:44.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/97698434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-133619-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:44.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:44.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:44.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:30:44.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:43 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:30:44.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:30:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:30:44.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:44.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/97698434' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-133619-20","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:44.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:44 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:44.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:44 smithi145 ceph-mon[95801]: osdmap e132: 8 total, 8 up, 8 in 2023-07-12T15:30:44.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-133849-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:44 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-133849-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:44.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:44 smithi145 ceph-mon[95801]: pgmap v142: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:44.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:44 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:45.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/97698434' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-133619-20","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[94241]: osdmap e132: 8 total, 8 up, 8 in 2023-07-12T15:30:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-133849-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-133849-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[94241]: pgmap v142: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:45.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:45.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/97698434' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-133619-20","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:45.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:45.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[102777]: osdmap e132: 8 total, 8 up, 8 in 2023-07-12T15:30:45.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-133849-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:45.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-133849-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:45.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[102777]: pgmap v142: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:45.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:44 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:45 smithi145 ceph-mon[95801]: 96.2 scrub starts 2023-07-12T15:30:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:45 smithi145 ceph-mon[95801]: 96.2 scrub ok 2023-07-12T15:30:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:45 smithi145 ceph-mon[95801]: osdmap e133: 8 total, 8 up, 8 in 2023-07-12T15:30:46.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3161262558' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-133627-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:46.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:46.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:45 smithi069 ceph-mon[102777]: 96.2 scrub starts 2023-07-12T15:30:46.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:45 smithi069 ceph-mon[102777]: 96.2 scrub ok 2023-07-12T15:30:46.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:45 smithi069 ceph-mon[102777]: osdmap e133: 8 total, 8 up, 8 in 2023-07-12T15:30:46.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3161262558' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-133627-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:46.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:46.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:45 smithi069 ceph-mon[94241]: 96.2 scrub starts 2023-07-12T15:30:46.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:45 smithi069 ceph-mon[94241]: 96.2 scrub ok 2023-07-12T15:30:46.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:45 smithi069 ceph-mon[94241]: osdmap e133: 8 total, 8 up, 8 in 2023-07-12T15:30:46.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3161262558' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-133627-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:46.150 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:30:46] "GET /metrics HTTP/1.1" 200 237830 "" "Prometheus/2.43.0" 2023-07-12T15:30:47.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:47.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-133849-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]': finished 2023-07-12T15:30:47.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3161262558' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-133627-17","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:47.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:46 smithi145 ceph-mon[95801]: osdmap e134: 8 total, 8 up, 8 in 2023-07-12T15:30:47.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/911771692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-133619-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:47.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-133619-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:47.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:46 smithi145 ceph-mon[95801]: pgmap v145: 364 pgs: 72 unknown, 292 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:47.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:47.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-133849-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]': finished 2023-07-12T15:30:47.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3161262558' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-133627-17","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:47.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[102777]: osdmap e134: 8 total, 8 up, 8 in 2023-07-12T15:30:47.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/911771692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-133619-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:47.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-133619-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:47.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[102777]: pgmap v145: 364 pgs: 72 unknown, 292 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:47.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:47.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi069-133849-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]': finished 2023-07-12T15:30:47.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3161262558' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-133627-17","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:47.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[94241]: osdmap e134: 8 total, 8 up, 8 in 2023-07-12T15:30:47.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/911771692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-133619-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:47.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-133619-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:47.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:46 smithi069 ceph-mon[94241]: pgmap v145: 364 pgs: 72 unknown, 292 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:48.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:48.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-133619-21","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:48.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:47 smithi145 ceph-mon[95801]: osdmap e135: 8 total, 8 up, 8 in 2023-07-12T15:30:48.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:48.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-133619-21","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:48.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:47 smithi069 ceph-mon[94241]: osdmap e135: 8 total, 8 up, 8 in 2023-07-12T15:30:48.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:48.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi069-133619-21","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:48.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:47 smithi069 ceph-mon[102777]: osdmap e135: 8 total, 8 up, 8 in 2023-07-12T15:30:49.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:49.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:48 smithi145 ceph-mon[95801]: osdmap e136: 8 total, 8 up, 8 in 2023-07-12T15:30:49.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/130108472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-133627-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:49.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:48 smithi145 ceph-mon[95801]: pgmap v148: 332 pgs: 40 unknown, 292 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:30:49.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:49.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:48 smithi069 ceph-mon[94241]: osdmap e136: 8 total, 8 up, 8 in 2023-07-12T15:30:49.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/130108472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-133627-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:49.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:48 smithi069 ceph-mon[94241]: pgmap v148: 332 pgs: 40 unknown, 292 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:30:49.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:48 smithi069 ceph-mon[102777]: osdmap e136: 8 total, 8 up, 8 in 2023-07-12T15:30:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/130108472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-133627-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:48 smithi069 ceph-mon[102777]: pgmap v148: 332 pgs: 40 unknown, 292 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:30:50.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:50.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/130108472' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-133627-18","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:50.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:49 smithi145 ceph-mon[95801]: osdmap e137: 8 total, 8 up, 8 in 2023-07-12T15:30:50.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2322161444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-133619-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:50.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:49 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-133619-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:50.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:50.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/130108472' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-133627-18","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:50.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:49 smithi069 ceph-mon[102777]: osdmap e137: 8 total, 8 up, 8 in 2023-07-12T15:30:50.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2322161444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-133619-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:50.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:49 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-133619-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:50.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:50.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/130108472' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi069-133627-18","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:49 smithi069 ceph-mon[94241]: osdmap e137: 8 total, 8 up, 8 in 2023-07-12T15:30:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2322161444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-133619-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:49 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-133619-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:51.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:51.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-133619-22","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:51.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:50 smithi145 ceph-mon[95801]: osdmap e138: 8 total, 8 up, 8 in 2023-07-12T15:30:51.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:50 smithi145 ceph-mon[95801]: pgmap v151: 332 pgs: 32 creating+peering, 300 active+clean; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:30:51.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-133619-22","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:50 smithi069 ceph-mon[102777]: osdmap e138: 8 total, 8 up, 8 in 2023-07-12T15:30:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:50 smithi069 ceph-mon[102777]: pgmap v151: 332 pgs: 32 creating+peering, 300 active+clean; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:30:51.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:51.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi069-133619-22","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:51.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:50 smithi069 ceph-mon[94241]: osdmap e138: 8 total, 8 up, 8 in 2023-07-12T15:30:51.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:50 smithi069 ceph-mon[94241]: pgmap v151: 332 pgs: 32 creating+peering, 300 active+clean; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:30:52.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:52.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:51 smithi145 ceph-mon[95801]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:52.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:51 smithi145 ceph-mon[95801]: osdmap e139: 8 total, 8 up, 8 in 2023-07-12T15:30:52.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/389141843' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-133627-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-133627-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-12T15:30:52.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-133627-19","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:52.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app1","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:52.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:51 smithi145 ceph-mon[95801]: osdmap e140: 8 total, 8 up, 8 in 2023-07-12T15:30:52.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2506097404' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi069-133619-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:52.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[94241]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:52.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[94241]: osdmap e139: 8 total, 8 up, 8 in 2023-07-12T15:30:52.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/389141843' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-133627-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-133627-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-12T15:30:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-133627-19","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app1","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[94241]: osdmap e140: 8 total, 8 up, 8 in 2023-07-12T15:30:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2506097404' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi069-133619-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[102777]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[102777]: osdmap e139: 8 total, 8 up, 8 in 2023-07-12T15:30:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/389141843' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-133627-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-133627-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-12T15:30:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi069-133627-19","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app1","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[102777]: osdmap e140: 8 total, 8 up, 8 in 2023-07-12T15:30:52.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2506097404' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi069-133619-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:53.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app2"}]: dispatch 2023-07-12T15:30:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:52 smithi145 ceph-mon[95801]: pgmap v154: 364 pgs: 64 unknown, 300 active+clean; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:30:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2506097404' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi069-133619-23","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:53.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app2","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:53.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:52 smithi145 ceph-mon[95801]: osdmap e141: 8 total, 8 up, 8 in 2023-07-12T15:30:53.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2023-07-12T15:30:53.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2023-07-12T15:30:53.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2023-07-12T15:30:53.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:53.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app2"}]: dispatch 2023-07-12T15:30:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[102777]: pgmap v154: 364 pgs: 64 unknown, 300 active+clean; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:30:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2506097404' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi069-133619-23","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app2","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:53.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[102777]: osdmap e141: 8 total, 8 up, 8 in 2023-07-12T15:30:53.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2023-07-12T15:30:53.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2023-07-12T15:30:53.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2023-07-12T15:30:53.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:53.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app2"}]: dispatch 2023-07-12T15:30:53.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:53.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:53.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[94241]: pgmap v154: 364 pgs: 64 unknown, 300 active+clean; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:30:53.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2506097404' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi069-133619-23","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:53.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi069-133766-1","app": "app2","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:53.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[94241]: osdmap e141: 8 total, 8 up, 8 in 2023-07-12T15:30:53.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2023-07-12T15:30:53.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2023-07-12T15:30:53.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2023-07-12T15:30:53.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:54.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:30:54 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:30:54.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1","value":"value1"}]': finished 2023-07-12T15:30:54.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:54 smithi145 ceph-mon[95801]: osdmap e142: 8 total, 8 up, 8 in 2023-07-12T15:30:54.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/53914055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi069-133627-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:54.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2023-07-12T15:30:54.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2023-07-12T15:30:54.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:54.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:54 smithi145 ceph-mon[95801]: pgmap v157: 332 pgs: 32 unknown, 300 active+clean; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:54.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:54 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:55.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1","value":"value1"}]': finished 2023-07-12T15:30:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[102777]: osdmap e142: 8 total, 8 up, 8 in 2023-07-12T15:30:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/53914055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi069-133627-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2023-07-12T15:30:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2023-07-12T15:30:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[102777]: pgmap v157: 332 pgs: 32 unknown, 300 active+clean; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1","value":"value1"}]': finished 2023-07-12T15:30:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[94241]: osdmap e142: 8 total, 8 up, 8 in 2023-07-12T15:30:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/53914055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi069-133627-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:55.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2023-07-12T15:30:55.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2023-07-12T15:30:55.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:55.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[94241]: pgmap v157: 332 pgs: 32 unknown, 300 active+clean; 217 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:55.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:54 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:30:56.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/53914055' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi069-133627-20","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:56.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:55 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key2","value":"value2"}]': finished 2023-07-12T15:30:56.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:55 smithi145 ceph-mon[95801]: osdmap e143: 8 total, 8 up, 8 in 2023-07-12T15:30:56.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2149332942' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-133619-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:56.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1"}]: dispatch 2023-07-12T15:30:56.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:55 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1"}]: dispatch 2023-07-12T15:30:56.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:56.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/53914055' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi069-133627-20","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:56.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key2","value":"value2"}]': finished 2023-07-12T15:30:56.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[102777]: osdmap e143: 8 total, 8 up, 8 in 2023-07-12T15:30:56.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2149332942' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-133619-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:56.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1"}]: dispatch 2023-07-12T15:30:56.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1"}]: dispatch 2023-07-12T15:30:56.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:56.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/53914055' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi069-133627-20","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:56.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key2","value":"value2"}]': finished 2023-07-12T15:30:56.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[94241]: osdmap e143: 8 total, 8 up, 8 in 2023-07-12T15:30:56.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2149332942' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-133619-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:56.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/886707965' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1"}]: dispatch 2023-07-12T15:30:56.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1"}]: dispatch 2023-07-12T15:30:56.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:56.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:30:56] "GET /metrics HTTP/1.1" 200 237830 "" "Prometheus/2.43.0" 2023-07-12T15:30:56.807 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-07-12T15:30:56.807 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-07-12T15:30:56.807 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-07-12T15:30:56.807 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-07-12T15:30:56.807 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-07-12T15:30:56.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-07-12T15:30:56.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-07-12T15:30:56.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-07-12T15:30:56.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: seed 133766 2023-07-12T15:30:56.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-07-12T15:30:56.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (8 ms) 2023-07-12T15:30:56.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-07-12T15:30:56.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (270 ms) 2023-07-12T15:30:56.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-07-12T15:30:56.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (8 ms) 2023-07-12T15:30:56.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-07-12T15:30:56.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (17 ms) 2023-07-12T15:30:56.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-07-12T15:30:56.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (6 ms) 2023-07-12T15:30:56.809 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-07-12T15:30:56.810 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (4 ms) 2023-07-12T15:30:56.810 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-07-12T15:30:56.810 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2023-07-12T15:30:56.810 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-07-12T15:30:56.810 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (4 ms) 2023-07-12T15:30:56.810 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-07-12T15:30:56.810 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2023-07-12T15:30:56.811 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-07-12T15:30:56.811 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (8 ms) 2023-07-12T15:30:56.811 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-07-12T15:30:56.811 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2023-07-12T15:30:56.811 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-07-12T15:30:56.811 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (5 ms) 2023-07-12T15:30:56.811 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-07-12T15:30:56.811 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (8 ms) 2023-07-12T15:30:56.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-07-12T15:30:56.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: osd_max_attr_size = 0 2023-07-12T15:30:56.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-07-12T15:30:56.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1697 ms) 2023-07-12T15:30:56.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-07-12T15:30:56.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (437 ms) 2023-07-12T15:30:56.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-07-12T15:30:56.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-07-12T15:30:56.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: done waiting, doing copies 2023-07-12T15:30:56.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: waiting for final deep scrubs... 2023-07-12T15:30:56.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: done waiting 2023-07-12T15:30:56.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60817 ms) 2023-07-12T15:30:56.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-07-12T15:30:56.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (33 ms) 2023-07-12T15:30:56.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-07-12T15:30:56.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (3 ms) 2023-07-12T15:30:56.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-07-12T15:30:56.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4921 ms) 2023-07-12T15:30:56.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-07-12T15:30:56.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-07-12T15:30:56.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-07-12T15:30:56.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-07-12T15:30:56.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-07-12T15:30:56.815 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2023-07-12T15:30:57.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2149332942' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-133619-24","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:57.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:56 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1"}]': finished 2023-07-12T15:30:57.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:56 smithi145 ceph-mon[95801]: osdmap e144: 8 total, 8 up, 8 in 2023-07-12T15:30:57.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:57.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:57.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:56 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:57.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:56 smithi145 ceph-mon[95801]: pgmap v160: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 1016 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:30:57.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2149332942' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-133619-24","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1"}]': finished 2023-07-12T15:30:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[102777]: osdmap e144: 8 total, 8 up, 8 in 2023-07-12T15:30:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:57.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[102777]: pgmap v160: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 1016 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:30:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2149332942' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi069-133619-24","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi069-133766-1","app":"app1","key":"key1"}]': finished 2023-07-12T15:30:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[94241]: osdmap e144: 8 total, 8 up, 8 in 2023-07-12T15:30:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:57.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:57.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:57.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:56 smithi069 ceph-mon[94241]: pgmap v160: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 1016 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:30:58.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:58.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:58.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]': finished 2023-07-12T15:30:58.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: osdmap e145: 8 total, 8 up, 8 in 2023-07-12T15:30:58.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:58.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:58.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115944138' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-133627-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:58.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-133627-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:58.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:58.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:58.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:58.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:58.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133766-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:58.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133766-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:58.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]': finished 2023-07-12T15:30:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: osdmap e145: 8 total, 8 up, 8 in 2023-07-12T15:30:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115944138' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-133627-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-133627-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:58.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:58.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133766-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:58.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133766-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:58.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:58.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:30:58.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:58.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi069-133849-15"}]': finished 2023-07-12T15:30:58.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: osdmap e145: 8 total, 8 up, 8 in 2023-07-12T15:30:58.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3899004439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:58.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]: dispatch 2023-07-12T15:30:58.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115944138' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-133627-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:58.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-133627-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:58.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:58.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:58.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:58.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:58.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133766-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:58.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133766-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:58.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]': finished 2023-07-12T15:30:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-133627-21","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133766-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: osdmap e146: 8 total, 8 up, 8 in 2023-07-12T15:30:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133766-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3426882049' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-133619-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133766-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:30:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:30:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:30:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:30:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:59.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:59.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: pgmap v163: 324 pgs: 64 unknown, 260 active+clean; 592 KiB data, 1016 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:59.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:30:59.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:59.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3426882049' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-133619-25","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:59.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:30:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:59.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]': finished 2023-07-12T15:30:59.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-133627-21","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:59.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133766-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:59.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: osdmap e146: 8 total, 8 up, 8 in 2023-07-12T15:30:59.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133766-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:59.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3426882049' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-133619-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:59.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133766-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:59.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:30:59.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:30:59.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:30:59.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:30:59.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:59.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:59.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: pgmap v163: 324 pgs: 64 unknown, 260 active+clean; 592 KiB data, 1016 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:59.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:30:59.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:59.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3426882049' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-133619-25","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:59.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:59.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi069-133849-15"}]': finished 2023-07-12T15:30:59.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi069-133627-21","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:59.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133766-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:30:59.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: osdmap e146: 8 total, 8 up, 8 in 2023-07-12T15:30:59.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133766-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:59.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3426882049' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-133619-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:30:59.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133766-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:30:59.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:30:59.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:30:59.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:30:59.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:30:59.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:59.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:30:59.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: pgmap v163: 324 pgs: 64 unknown, 260 active+clean; 592 KiB data, 1016 MiB used, 714 GiB / 715 GiB avail 2023-07-12T15:30:59.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:30:59.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:30:59.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3426882049' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi069-133619-25","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:30:59.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:30:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:00.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP api_snapshots_pp: Running main() from gmock_main.cc 2023-07-12T15:31:00.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-07-12T15:31:00.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-07-12T15:31:00.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-07-12T15:31:00.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: seed 133849 2023-07-12T15:31:00.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-07-12T15:31:00.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2857 ms) 2023-07-12T15:31:00.815 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-07-12T15:31:00.815 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (1536 ms) 2023-07-12T15:31:00.815 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-07-12T15:31:00.815 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2276 ms) 2023-07-12T15:31:00.815 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-07-12T15:31:00.815 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (1752 ms) 2023-07-12T15:31:00.815 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-07-12T15:31:00.816 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3559 ms) 2023-07-12T15:31:00.816 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11980 ms total) 2023-07-12T15:31:00.816 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-07-12T15:31:00.816 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-07-12T15:31:00.816 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-07-12T15:31:00.816 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4050 ms) 2023-07-12T15:31:00.817 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-07-12T15:31:00.817 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (5105 ms) 2023-07-12T15:31:00.817 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-07-12T15:31:00.817 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6117 ms) 2023-07-12T15:31:00.817 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-07-12T15:31:00.817 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3653 ms) 2023-07-12T15:31:00.817 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-07-12T15:31:00.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2173 ms) 2023-07-12T15:31:00.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-07-12T15:31:00.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.1.2-55-gb1bd0540/rpm/el8/BUILD/ceph-18.1.2-55-gb1bd0540/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-07-12T15:31:00.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-07-12T15:31:00.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-07-12T15:31:00.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-07-12T15:31:00.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi069-133849-7 2023-07-12T15:31:00.818 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: waiting for snaps to purge 2023-07-12T15:31:00.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17970 ms) 2023-07-12T15:31:00.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (39068 ms total) 2023-07-12T15:31:00.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-07-12T15:31:00.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-07-12T15:31:00.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-07-12T15:31:00.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3011 ms) 2023-07-12T15:31:00.819 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-07-12T15:31:00.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2013 ms) 2023-07-12T15:31:00.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-07-12T15:31:00.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (1912 ms) 2023-07-12T15:31:00.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-07-12T15:31:00.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2004 ms) 2023-07-12T15:31:00.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8940 ms total) 2023-07-12T15:31:00.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-07-12T15:31:00.820 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-07-12T15:31:00.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:00 smithi145 ceph-mon[95801]: osdmap e147: 8 total, 8 up, 8 in 2023-07-12T15:31:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:00 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:00 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133766-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133766-24"}]': finished 2023-07-12T15:31:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:00 smithi145 ceph-mon[95801]: osdmap e148: 8 total, 8 up, 8 in 2023-07-12T15:31:00.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:00 smithi069 ceph-mon[102777]: osdmap e147: 8 total, 8 up, 8 in 2023-07-12T15:31:00.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:00.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:00 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:00.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:00.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:00 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133766-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133766-24"}]': finished 2023-07-12T15:31:00.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:00 smithi069 ceph-mon[102777]: osdmap e148: 8 total, 8 up, 8 in 2023-07-12T15:31:00.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:00 smithi069 ceph-mon[94241]: osdmap e147: 8 total, 8 up, 8 in 2023-07-12T15:31:00.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:00.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:00 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:00.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:00.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:00 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133766-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133766-24"}]': finished 2023-07-12T15:31:00.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:00 smithi069 ceph-mon[94241]: osdmap e148: 8 total, 8 up, 8 in 2023-07-12T15:31:01.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:01 smithi145 ceph-mon[95801]: pgmap v166: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 368 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:01.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1521452706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-133627-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:01.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:01 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]': finished 2023-07-12T15:31:01.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1521452706' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-133627-22","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:01.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:01 smithi145 ceph-mon[95801]: osdmap e149: 8 total, 8 up, 8 in 2023-07-12T15:31:01.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:01 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2600887021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-133619-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:01 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-133619-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[102777]: pgmap v166: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 368 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:01.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1521452706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-133627-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:01.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]': finished 2023-07-12T15:31:01.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1521452706' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-133627-22","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:01.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[102777]: osdmap e149: 8 total, 8 up, 8 in 2023-07-12T15:31:01.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2600887021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-133619-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-133619-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[94241]: pgmap v166: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 368 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:01.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1521452706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-133627-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:01.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]': finished 2023-07-12T15:31:01.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1521452706' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi069-133627-22","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:01.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[94241]: osdmap e149: 8 total, 8 up, 8 in 2023-07-12T15:31:01.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2600887021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-133619-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:01.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:01 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-133619-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:03.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:02 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-24","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:02 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-133619-26","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:02 smithi145 ceph-mon[95801]: osdmap e150: 8 total, 8 up, 8 in 2023-07-12T15:31:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:02 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:02 smithi145 ceph-mon[95801]: pgmap v169: 340 pgs: 80 unknown, 260 active+clean; 592 KiB data, 368 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:03.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-24","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-133619-26","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[94241]: osdmap e150: 8 total, 8 up, 8 in 2023-07-12T15:31:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[94241]: pgmap v169: 340 pgs: 80 unknown, 260 active+clean; 592 KiB data, 368 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-24","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi069-133619-26","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[102777]: osdmap e150: 8 total, 8 up, 8 in 2023-07-12T15:31:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:03.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:02 smithi069 ceph-mon[102777]: pgmap v169: 340 pgs: 80 unknown, 260 active+clean; 592 KiB data, 368 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:04.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:04.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:03 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-25","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:04.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:03 smithi145 ceph-mon[95801]: osdmap e151: 8 total, 8 up, 8 in 2023-07-12T15:31:04.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/757716854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-133627-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:04.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:03 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-133627-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:04.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:03 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:03 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-133619-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:04.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:03 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-133619-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:04.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:04.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-25","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:04.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[94241]: osdmap e151: 8 total, 8 up, 8 in 2023-07-12T15:31:04.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/757716854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-133627-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:04.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-133627-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:04.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-133619-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:04.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-133619-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:04.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:04.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133766-25","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:04.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[102777]: osdmap e151: 8 total, 8 up, 8 in 2023-07-12T15:31:04.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/757716854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-133627-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:04.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-133627-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:04.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-133619-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:04.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:03 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-133619-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:04.851 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:31:04 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:31:04.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:04.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:04 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-133627-23","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:04.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:04 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-133619-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:04.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:04 smithi145 ceph-mon[95801]: osdmap e152: 8 total, 8 up, 8 in 2023-07-12T15:31:04.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-133619-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:04 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-133619-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:04.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:04 smithi145 ceph-mon[95801]: pgmap v172: 308 pgs: 48 unknown, 260 active+clean; 592 KiB data, 368 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:04.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:04 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:05.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:05.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-133627-23","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:05.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-133619-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[94241]: osdmap e152: 8 total, 8 up, 8 in 2023-07-12T15:31:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-133619-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-133619-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:05.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[94241]: pgmap v172: 308 pgs: 48 unknown, 260 active+clean; 592 KiB data, 368 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:05.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:05.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:05.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi069-133627-23","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:05.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi069-133619-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:05.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[102777]: osdmap e152: 8 total, 8 up, 8 in 2023-07-12T15:31:05.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-133619-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:05.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-133619-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:05.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[102777]: pgmap v172: 308 pgs: 48 unknown, 260 active+clean; 592 KiB data, 368 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:05.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:04 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:06.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:06.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:05 smithi145 ceph-mon[95801]: osdmap e153: 8 total, 8 up, 8 in 2023-07-12T15:31:06.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:31:06.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:05 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:31:06.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:06.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:05 smithi069 ceph-mon[102777]: osdmap e153: 8 total, 8 up, 8 in 2023-07-12T15:31:06.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:31:06.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:05 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:31:06.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:06.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:05 smithi069 ceph-mon[94241]: osdmap e153: 8 total, 8 up, 8 in 2023-07-12T15:31:06.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:31:06.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:05 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:31:06.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:31:06] "GET /metrics HTTP/1.1" 200 235436 "" "Prometheus/2.43.0" 2023-07-12T15:31:07.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:07.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:06 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-133619-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-133619-27"}]': finished 2023-07-12T15:31:07.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:06 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]': finished 2023-07-12T15:31:07.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:06 smithi145 ceph-mon[95801]: osdmap e154: 8 total, 8 up, 8 in 2023-07-12T15:31:07.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2623160553' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-133627-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:07.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:31:07.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:06 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-133627-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:07.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:06 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:31:07.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:06 smithi145 ceph-mon[95801]: pgmap v175: 308 pgs: 40 unknown, 268 active+clean; 608 KiB data, 357 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2023-07-12T15:31:07.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:07.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-133619-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-133619-27"}]': finished 2023-07-12T15:31:07.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]': finished 2023-07-12T15:31:07.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[102777]: osdmap e154: 8 total, 8 up, 8 in 2023-07-12T15:31:07.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2623160553' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-133627-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:07.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:31:07.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-133627-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:07.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:31:07.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[102777]: pgmap v175: 308 pgs: 40 unknown, 268 active+clean; 608 KiB data, 357 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2023-07-12T15:31:07.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:07.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi069-133619-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi069-133619-27"}]': finished 2023-07-12T15:31:07.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133766-24"}]': finished 2023-07-12T15:31:07.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[94241]: osdmap e154: 8 total, 8 up, 8 in 2023-07-12T15:31:07.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2623160553' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-133627-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:07.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2135291342' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:31:07.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-133627-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:07.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]: dispatch 2023-07-12T15:31:07.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:06 smithi069 ceph-mon[94241]: pgmap v175: 308 pgs: 40 unknown, 268 active+clean; 608 KiB data, 357 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2023-07-12T15:31:08.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:08.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:07 smithi145 ceph-mon[95801]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:31:08.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:07 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-133627-24","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:08.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:07 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]': finished 2023-07-12T15:31:08.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:07 smithi145 ceph-mon[95801]: osdmap e155: 8 total, 8 up, 8 in 2023-07-12T15:31:08.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:08.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:07 smithi069 ceph-mon[102777]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:31:08.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:07 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-133627-24","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:08.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:07 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]': finished 2023-07-12T15:31:08.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:07 smithi069 ceph-mon[102777]: osdmap e155: 8 total, 8 up, 8 in 2023-07-12T15:31:08.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:08.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:07 smithi069 ceph-mon[94241]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:31:08.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:07 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi069-133627-24","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:08.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:07 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133766-24"}]': finished 2023-07-12T15:31:08.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:07 smithi069 ceph-mon[94241]: osdmap e155: 8 total, 8 up, 8 in 2023-07-12T15:31:09.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:08 smithi145 ceph-mon[95801]: osdmap e156: 8 total, 8 up, 8 in 2023-07-12T15:31:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:08 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1137212095' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi069-133766-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:08 smithi145 ceph-mon[95801]: pgmap v178: 300 pgs: 32 unknown, 268 active+clean; 608 KiB data, 357 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2023-07-12T15:31:09.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:09.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:08 smithi069 ceph-mon[94241]: osdmap e156: 8 total, 8 up, 8 in 2023-07-12T15:31:09.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:09.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:08 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:09.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1137212095' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi069-133766-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:09.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:08 smithi069 ceph-mon[94241]: pgmap v178: 300 pgs: 32 unknown, 268 active+clean; 608 KiB data, 357 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2023-07-12T15:31:09.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:09.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:08 smithi069 ceph-mon[102777]: osdmap e156: 8 total, 8 up, 8 in 2023-07-12T15:31:09.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:09.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:08 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:09.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1137212095' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi069-133766-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:09.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:08 smithi069 ceph-mon[102777]: pgmap v178: 300 pgs: 32 unknown, 268 active+clean; 608 KiB data, 357 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2023-07-12T15:31:09.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRad api_aio: Running main() from gmock_main.cc 2023-07-12T15:31:09.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-07-12T15:31:09.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] Global test environment set-up. 2023-07-12T15:31:09.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-07-12T15:31:09.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-07-12T15:31:09.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.TooBig (6940 ms) 2023-07-12T15:31:09.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-07-12T15:31:09.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3625 ms) 2023-07-12T15:31:09.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-07-12T15:31:09.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3011 ms) 2023-07-12T15:31:09.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-07-12T15:31:09.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (2336 ms) 2023-07-12T15:31:09.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-07-12T15:31:09.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3003 ms) 2023-07-12T15:31:09.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-07-12T15:31:09.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3255 ms) 2023-07-12T15:31:09.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-07-12T15:31:09.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3113 ms) 2023-07-12T15:31:09.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-07-12T15:31:09.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2848 ms) 2023-07-12T15:31:09.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-07-12T15:31:09.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3239 ms) 2023-07-12T15:31:09.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-07-12T15:31:09.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3026 ms) 2023-07-12T15:31:09.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-07-12T15:31:09.870 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3084 ms) 2023-07-12T15:31:09.870 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-07-12T15:31:09.870 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2988 ms) 2023-07-12T15:31:09.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-07-12T15:31:09.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2727 ms) 2023-07-12T15:31:09.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-07-12T15:31:09.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3063 ms) 2023-07-12T15:31:09.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-07-12T15:31:09.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.Flush (2984 ms) 2023-07-12T15:31:09.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-07-12T15:31:09.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2887 ms) 2023-07-12T15:31:09.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-07-12T15:31:09.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2998 ms) 2023-07-12T15:31:09.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-07-12T15:31:09.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3024 ms) 2023-07-12T15:31:09.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-07-12T15:31:09.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3020 ms) 2023-07-12T15:31:09.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-07-12T15:31:09.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3237 ms) 2023-07-12T15:31:09.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-07-12T15:31:09.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (2786 ms) 2023-07-12T15:31:09.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-07-12T15:31:09.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3020 ms) 2023-07-12T15:31:09.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-07-12T15:31:09.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2916 ms) 2023-07-12T15:31:09.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-07-12T15:31:09.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3030 ms) 2023-07-12T15:31:09.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-07-12T15:31:09.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3235 ms) 2023-07-12T15:31:09.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-07-12T15:31:09.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (2781 ms) 2023-07-12T15:31:09.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 26 tests from LibRadosAio (82178 ms total) 2023-07-12T15:31:09.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: 2023-07-12T15:31:09.875 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-07-12T15:31:09.875 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2023-07-12T15:31:10.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:10.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:09 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]': finished 2023-07-12T15:31:10.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1137212095' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi069-133766-27","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:10.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:09 smithi145 ceph-mon[95801]: osdmap e157: 8 total, 8 up, 8 in 2023-07-12T15:31:10.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/774028175' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi069-133627-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:10.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:10.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:09 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:10.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:10.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]': finished 2023-07-12T15:31:10.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1137212095' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi069-133766-27","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:10.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[94241]: osdmap e157: 8 total, 8 up, 8 in 2023-07-12T15:31:10.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/774028175' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi069-133627-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:10.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:10.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:10.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:10.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi069-133619-27"}]': finished 2023-07-12T15:31:10.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1137212095' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi069-133766-27","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:10.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[102777]: osdmap e157: 8 total, 8 up, 8 in 2023-07-12T15:31:10.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/774028175' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi069-133627-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:10.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2242965564' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:10.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:09 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]: dispatch 2023-07-12T15:31:11.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:11.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/774028175' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi069-133627-25","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:11.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:10 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]': finished 2023-07-12T15:31:11.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:10 smithi145 ceph-mon[95801]: osdmap e158: 8 total, 8 up, 8 in 2023-07-12T15:31:11.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:11.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:10 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:11.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:11.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:10 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:11.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-133619-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:11.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:10 smithi145 ceph-mon[95801]: pgmap v181: 300 pgs: 32 creating+peering, 268 active+clean; 611 KiB data, 346 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 3 op/s 2023-07-12T15:31:11.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:10 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-133619-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:11.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/774028175' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi069-133627-25","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]': finished 2023-07-12T15:31:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[102777]: osdmap e158: 8 total, 8 up, 8 in 2023-07-12T15:31:11.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:11.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:11.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:11.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:11.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-133619-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:11.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[102777]: pgmap v181: 300 pgs: 32 creating+peering, 268 active+clean; 611 KiB data, 346 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 3 op/s 2023-07-12T15:31:11.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-133619-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:11.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:11.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:11.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/774028175' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi069-133627-25","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:11.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi069-133619-27"}]': finished 2023-07-12T15:31:11.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[94241]: osdmap e158: 8 total, 8 up, 8 in 2023-07-12T15:31:11.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:11.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:11.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:11.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:11.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-133619-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:11.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[94241]: pgmap v181: 300 pgs: 32 creating+peering, 268 active+clean; 611 KiB data, 346 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 3 op/s 2023-07-12T15:31:11.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-133619-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:11.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:12.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:12.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:12 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-133619-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:12.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:12.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:12 smithi145 ceph-mon[95801]: osdmap e159: 8 total, 8 up, 8 in 2023-07-12T15:31:12.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-13"}]: dispatch 2023-07-12T15:31:12.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-133619-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:12.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:12 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-133619-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:12.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2014842989' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-133766-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:12.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:12 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-133766-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:12.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:12 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:12.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-13"}]': finished 2023-07-12T15:31:12.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:12.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-133619-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:12.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:12.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[102777]: osdmap e159: 8 total, 8 up, 8 in 2023-07-12T15:31:12.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-13"}]: dispatch 2023-07-12T15:31:12.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-133619-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:12.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-133619-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:12.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2014842989' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-133766-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:12.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-133766-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:12.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:12.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-13"}]': finished 2023-07-12T15:31:12.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:12.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi069-133619-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:12.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:12.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[94241]: osdmap e159: 8 total, 8 up, 8 in 2023-07-12T15:31:12.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-13"}]: dispatch 2023-07-12T15:31:12.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-133619-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:12.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-133619-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:12.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2014842989' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-133766-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:12.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-133766-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:12.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:12.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-13"}]': finished 2023-07-12T15:31:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:13 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-133766-30","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:13 smithi145 ceph-mon[95801]: osdmap e160: 8 total, 8 up, 8 in 2023-07-12T15:31:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:13 smithi145 ceph-mon[95801]: pgmap v184: 332 pgs: 64 unknown, 268 active+clean; 611 KiB data, 346 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 3 op/s 2023-07-12T15:31:13.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2932111032' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-133627-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:13.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:13 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-133627-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:13.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:13 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-133619-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-133619-28"}]': finished 2023-07-12T15:31:13.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:13 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-133627-26","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:13.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:13 smithi145 ceph-mon[95801]: osdmap e161: 8 total, 8 up, 8 in 2023-07-12T15:31:13.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:13.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-133766-30","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:13.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[102777]: osdmap e160: 8 total, 8 up, 8 in 2023-07-12T15:31:13.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[102777]: pgmap v184: 332 pgs: 64 unknown, 268 active+clean; 611 KiB data, 346 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 3 op/s 2023-07-12T15:31:13.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2932111032' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-133627-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:13.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-133627-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:13.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-133619-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-133619-28"}]': finished 2023-07-12T15:31:13.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-133627-26","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:13.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[102777]: osdmap e161: 8 total, 8 up, 8 in 2023-07-12T15:31:13.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:13.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi069-133766-30","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:13.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[94241]: osdmap e160: 8 total, 8 up, 8 in 2023-07-12T15:31:13.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[94241]: pgmap v184: 332 pgs: 64 unknown, 268 active+clean; 611 KiB data, 346 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 3 op/s 2023-07-12T15:31:13.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2932111032' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-133627-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:13.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-133627-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:13.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi069-133619-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi069-133619-28"}]': finished 2023-07-12T15:31:13.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi069-133627-26","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:13.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:13 smithi069 ceph-mon[94241]: osdmap e161: 8 total, 8 up, 8 in 2023-07-12T15:31:14.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:14 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:31:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:14 smithi145 ceph-mon[95801]: osdmap e162: 8 total, 8 up, 8 in 2023-07-12T15:31:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2973983284' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-133766-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-133766-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:14.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:31:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[102777]: osdmap e162: 8 total, 8 up, 8 in 2023-07-12T15:31:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2973983284' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-133766-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:14.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-133766-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:14.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:14.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:14.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:14.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:14.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:31:14.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[94241]: osdmap e162: 8 total, 8 up, 8 in 2023-07-12T15:31:14.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:14.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2973983284' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-133766-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:14.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:14.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-133766-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:14.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:14.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:14.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:31:14 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:31:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: pgmap v187: 300 pgs: 72 unknown, 228 active+clean; 586 KiB data, 346 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:31:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:15.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:31:15.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]': finished 2023-07-12T15:31:15.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-133766-33","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:15.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-15","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:15.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: osdmap e163: 8 total, 8 up, 8 in 2023-07-12T15:31:15.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:15.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:15.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/286602578' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-133627-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:15.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:15.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:15.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:15.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: pgmap v187: 300 pgs: 72 unknown, 228 active+clean; 586 KiB data, 346 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:31:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:31:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]': finished 2023-07-12T15:31:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-133766-33","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-15","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: osdmap e163: 8 total, 8 up, 8 in 2023-07-12T15:31:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/286602578' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-133627-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:15.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: pgmap v187: 300 pgs: 72 unknown, 228 active+clean; 586 KiB data, 346 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:31:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:31:15.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]': finished 2023-07-12T15:31:15.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi069-133766-33","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:15.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-15","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:15.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: osdmap e163: 8 total, 8 up, 8 in 2023-07-12T15:31:15.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1713693570' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:15.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]: dispatch 2023-07-12T15:31:15.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/286602578' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-133627-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:15.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:15.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:15.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: osSnapshotsSelfManagedECPP.SnapPP 2023-07-12T15:31:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4258 ms) 2023-07-12T15:31:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-07-12T15:31:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3793 ms) 2023-07-12T15:31:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-07-12T15:31:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3907 ms) 2023-07-12T15:31:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11958 ms total) 2023-07-12T15:31:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2023-07-12T15:31:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-07-12T15:31:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (95039 ms total) 2023-07-12T15:31:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-07-12T15:31:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-07-12T15:31:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-07-12T15:31:15.796 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:31:15.796 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133739 2023-07-12T15:31:15.796 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133739 2023-07-12T15:31:15.796 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:31:15.796 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133701 2023-07-12T15:31:15.796 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133701 2023-07-12T15:31:15.797 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:31:15.797 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133654 2023-07-12T15:31:15.797 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133654 2023-07-12T15:31:16.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:31:16] "GET /metrics HTTP/1.1" 200 235473 "" "Prometheus/2.43.0" 2023-07-12T15:31:17.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:16 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]': finished 2023-07-12T15:31:17.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/286602578' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-133627-27","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:17.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:16 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]': finished 2023-07-12T15:31:17.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:16 smithi145 ceph-mon[95801]: osdmap e164: 8 total, 8 up, 8 in 2023-07-12T15:31:17.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:17.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:16 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:17.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-15", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:17.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:17.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:17.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:17.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:17.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:16 smithi145 ceph-mon[95801]: pgmap v190: 292 pgs: 32 creating+peering, 32 unknown, 228 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:17.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]': finished 2023-07-12T15:31:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/286602578' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-133627-27","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]': finished 2023-07-12T15:31:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[102777]: osdmap e164: 8 total, 8 up, 8 in 2023-07-12T15:31:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-15", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[102777]: pgmap v190: 292 pgs: 32 creating+peering, 32 unknown, 228 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi069-133849-20"}]': finished 2023-07-12T15:31:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/286602578' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi069-133627-27","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi069-133619-28"}]': finished 2023-07-12T15:31:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[94241]: osdmap e164: 8 total, 8 up, 8 in 2023-07-12T15:31:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2903439300' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]: dispatch 2023-07-12T15:31:17.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-15", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:17.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:17.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:17.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:17.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:17.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:16 smithi069 ceph-mon[94241]: pgmap v190: 292 pgs: 32 creating+peering, 32 unknown, 228 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:18.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:17 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]': finished 2023-07-12T15:31:18.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-15", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:18.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:18.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:17 smithi145 ceph-mon[95801]: osdmap e165: 8 total, 8 up, 8 in 2023-07-12T15:31:18.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-133766-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:18.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-15"}]: dispatch 2023-07-12T15:31:18.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:18.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:18.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:17 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:18.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:18.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:17 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:18.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-133619-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:18.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:17 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-133619-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:18.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]': finished 2023-07-12T15:31:18.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-15", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:18.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:18.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[102777]: osdmap e165: 8 total, 8 up, 8 in 2023-07-12T15:31:18.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-133766-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:18.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-15"}]: dispatch 2023-07-12T15:31:18.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:18.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:18.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:18.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:18.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:18.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-133619-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:18.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-133619-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:18.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi069-133619-28"}]': finished 2023-07-12T15:31:18.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-15", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:18.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:18.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[94241]: osdmap e165: 8 total, 8 up, 8 in 2023-07-12T15:31:18.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-133766-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:18.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-15"}]: dispatch 2023-07-12T15:31:18.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:18.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:18.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:18.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:18.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:18.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-133619-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:18.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:17 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-133619-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:19.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:19.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:18 smithi145 ceph-mon[95801]: pgmap v193: 292 pgs: 32 unknown, 32 creating+peering, 228 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:19.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-15"}]': finished 2023-07-12T15:31:19.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:18 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-133619-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:19.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:18 smithi145 ceph-mon[95801]: osdmap e166: 8 total, 8 up, 8 in 2023-07-12T15:31:19.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-133619-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:19.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-15", "mode": "writeback"}]: dispatch 2023-07-12T15:31:19.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/394314755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-133627-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:19.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:18 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-133619-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:19.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:18 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:19.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:18 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:19.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:19.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[94241]: pgmap v193: 292 pgs: 32 unknown, 32 creating+peering, 228 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:19.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-15"}]': finished 2023-07-12T15:31:19.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-133619-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:19.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[94241]: osdmap e166: 8 total, 8 up, 8 in 2023-07-12T15:31:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-133619-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-15", "mode": "writeback"}]: dispatch 2023-07-12T15:31:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/394314755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-133627-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-133619-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:19.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:19.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[102777]: pgmap v193: 292 pgs: 32 unknown, 32 creating+peering, 228 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:19.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-15"}]': finished 2023-07-12T15:31:19.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi069-133619-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:19.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[102777]: osdmap e166: 8 total, 8 up, 8 in 2023-07-12T15:31:19.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-133619-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:19.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-15", "mode": "writeback"}]: dispatch 2023-07-12T15:31:19.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/394314755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-133627-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:19.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-133619-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:19.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:19.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:18 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:20.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:19 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:20.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-133766-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-133766-36"}]': finished 2023-07-12T15:31:20.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-15", "mode": "writeback"}]': finished 2023-07-12T15:31:20.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/394314755' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-133627-28","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:20.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:20.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:19 smithi145 ceph-mon[95801]: osdmap e167: 8 total, 8 up, 8 in 2023-07-12T15:31:20.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:19 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:20.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:19 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:20.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:20.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-133766-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-133766-36"}]': finished 2023-07-12T15:31:20.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-15", "mode": "writeback"}]': finished 2023-07-12T15:31:20.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/394314755' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-133627-28","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:20.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:20.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[102777]: osdmap e167: 8 total, 8 up, 8 in 2023-07-12T15:31:20.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:20.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:20.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:20.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi069-133766-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi069-133766-36"}]': finished 2023-07-12T15:31:20.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-15", "mode": "writeback"}]': finished 2023-07-12T15:31:20.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/394314755' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi069-133627-28","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:20.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:20.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[94241]: osdmap e167: 8 total, 8 up, 8 in 2023-07-12T15:31:20.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:20.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:19 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:21.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:20 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-133619-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-133619-29"}]': finished 2023-07-12T15:31:21.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:20 smithi145 ceph-mon[95801]: osdmap e168: 8 total, 8 up, 8 in 2023-07-12T15:31:21.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:20 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:31:21.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:20 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:31:21.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:21.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:20 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:21.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:20 smithi145 ceph-mon[95801]: pgmap v196: 276 pgs: 8 creating+peering, 8 unknown, 260 active+clean; 585 KiB data, 356 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:31:21.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:21.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-133619-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-133619-29"}]': finished 2023-07-12T15:31:21.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[102777]: osdmap e168: 8 total, 8 up, 8 in 2023-07-12T15:31:21.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:31:21.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:31:21.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:21.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:21.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[102777]: pgmap v196: 276 pgs: 8 creating+peering, 8 unknown, 260 active+clean; 585 KiB data, 356 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:31:21.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:21.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi069-133619-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi069-133619-29"}]': finished 2023-07-12T15:31:21.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[94241]: osdmap e168: 8 total, 8 up, 8 in 2023-07-12T15:31:21.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:31:21.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:31:21.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:21.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:21.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[94241]: pgmap v196: 276 pgs: 8 creating+peering, 8 unknown, 260 active+clean; 585 KiB data, 356 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:31:21.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:22.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:22.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: osdmap e169: 8 total, 8 up, 8 in 2023-07-12T15:31:22.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-15"}]: dispatch 2023-07-12T15:31:22.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:22.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3635692204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-133627-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:22.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-133627-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:22.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:22.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:22.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-15"}]': finished 2023-07-12T15:31:22.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36"}]': finished 2023-07-12T15:31:22.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-133627-29","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:22.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: osdmap e170: 8 total, 8 up, 8 in 2023-07-12T15:31:22.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:22.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:22.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:21 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:22.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:22.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: osdmap e169: 8 total, 8 up, 8 in 2023-07-12T15:31:22.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-15"}]: dispatch 2023-07-12T15:31:22.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:22.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3635692204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-133627-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-133627-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-15"}]': finished 2023-07-12T15:31:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36"}]': finished 2023-07-12T15:31:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-133627-29","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:22.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: osdmap e170: 8 total, 8 up, 8 in 2023-07-12T15:31:22.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:22.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:22.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:22.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:22.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: osdmap e169: 8 total, 8 up, 8 in 2023-07-12T15:31:22.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-15"}]: dispatch 2023-07-12T15:31:22.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:22.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3635692204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-133627-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:22.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-133627-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:22.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:22.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:22.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-15"}]': finished 2023-07-12T15:31:22.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi069-133766-36"}]': finished 2023-07-12T15:31:22.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi069-133627-29","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:22.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: osdmap e170: 8 total, 8 up, 8 in 2023-07-12T15:31:22.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-133766-36"}]: dispatch 2023-07-12T15:31:22.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:22.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:21 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:22.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: Running main() from gmock_main.cc 2023-07-12T15:31:22.777 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-07-12T15:31:22.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-07-12T15:31:22.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-07-12T15:31:22.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-07-12T15:31:22.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (6902 ms) 2023-07-12T15:31:22.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-07-12T15:31:22.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (16345 ms) 2023-07-12T15:31:22.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-07-12T15:31:22.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (7058 ms) 2023-07-12T15:31:22.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-07-12T15:31:22.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3092 ms) 2023-07-12T15:31:22.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-07-12T15:31:22.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (2818 ms) 2023-07-12T15:31:22.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-07-12T15:31:22.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3272 ms) 2023-07-12T15:31:22.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-07-12T15:31:22.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2649 ms) 2023-07-12T15:31:22.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-07-12T15:31:22.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3099 ms) 2023-07-12T15:31:22.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-07-12T15:31:22.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3035 ms) 2023-07-12T15:31:22.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-07-12T15:31:22.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2857 ms) 2023-07-12T15:31:22.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-07-12T15:31:22.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3027 ms) 2023-07-12T15:31:22.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-07-12T15:31:22.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2984 ms) 2023-07-12T15:31:22.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-07-12T15:31:22.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3020 ms) 2023-07-12T15:31:22.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-07-12T15:31:22.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3026 ms) 2023-07-12T15:31:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-07-12T15:31:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3006 ms) 2023-07-12T15:31:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-07-12T15:31:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3008 ms) 2023-07-12T15:31:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-07-12T15:31:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2924 ms) 2023-07-12T15:31:22.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-07-12T15:31:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3013 ms) 2023-07-12T15:31:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-07-12T15:31:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3261 ms) 2023-07-12T15:31:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-07-12T15:31:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2787 ms) 2023-07-12T15:31:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-07-12T15:31:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3241 ms) 2023-07-12T15:31:22.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-07-12T15:31:22.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2800 ms) 2023-07-12T15:31:22.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-07-12T15:31:22.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3013 ms) 2023-07-12T15:31:22.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-07-12T15:31:22.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2892 ms) 2023-07-12T15:31:22.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-07-12T15:31:22.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3011 ms) 2023-07-12T15:31:22.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-07-12T15:31:22.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3012 ms) 2023-07-12T15:31:22.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-07-12T15:31:22.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio(68252 ms total) 2023-07-12T15:31:22.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-07-12T15:31:22.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-07-12T15:31:22.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-07-12T15:31:22.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (125 ms) 2023-07-12T15:31:22.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (125 ms total) 2023-07-12T15:31:22.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-07-12T15:31:22.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-07-12T15:31:22.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-07-12T15:31:22.787 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (17 ms) 2023-07-12T15:31:22.787 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-07-12T15:31:22.787 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (5 ms) 2023-07-12T15:31:22.787 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (22 ms total) 2023-07-12T15:31:22.787 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-07-12T15:31:22.787 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-07-12T15:31:22.787 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-07-12T15:31:22.787 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (5 ms) 2023-07-12T15:31:22.788 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-07-12T15:31:22.788 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (5 ms) 2023-07-12T15:31:22.788 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (10 ms total) 2023-07-12T15:31:22.788 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-07-12T15:31:22.788 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-07-12T15:31:22.788 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-07-12T15:31:22.788 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (16 ms) 2023-07-12T15:31:22.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-07-12T15:31:22.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (11 ms) 2023-07-12T15:31:22.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (27 ms total) 2023-07-12T15:31:22.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-07-12T15:31:22.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-07-12T15:31:22.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-07-12T15:31:22.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (787 ms) 2023-07-12T15:31:22.789 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (787 ms total) 2023-07-12T15:31:22.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2023-07-12T15:31:22.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-07-12T15:31:22.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (102043 ms total) 2023-07-12T15:31:22.790 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ PASSED ] 31 tests. 2023-07-12T15:31:23.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:23 smithi145 ceph-mon[95801]: pgmap v199: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 356 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:31:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-133766-36"}]': finished 2023-07-12T15:31:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:23 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]': finished 2023-07-12T15:31:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:23 smithi145 ceph-mon[95801]: osdmap e171: 8 total, 8 up, 8 in 2023-07-12T15:31:23.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:23.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:23 smithi069 ceph-mon[94241]: pgmap v199: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 356 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:31:23.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-133766-36"}]': finished 2023-07-12T15:31:23.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:23 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]': finished 2023-07-12T15:31:23.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:23 smithi069 ceph-mon[94241]: osdmap e171: 8 total, 8 up, 8 in 2023-07-12T15:31:23.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:23.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:23 smithi069 ceph-mon[102777]: pgmap v199: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 356 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:31:23.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/775465964' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi069-133766-36"}]': finished 2023-07-12T15:31:23.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:23 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi069-133619-29"}]': finished 2023-07-12T15:31:23.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:23 smithi069 ceph-mon[102777]: osdmap e171: 8 total, 8 up, 8 in 2023-07-12T15:31:24.573 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:24.574 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:24.574 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:24.574 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]': finished 2023-07-12T15:31:24.575 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: osdmap e172: 8 total, 8 up, 8 in 2023-07-12T15:31:24.575 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2701312756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-133627-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:24.575 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:24.575 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-133627-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:24.576 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:24.576 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:24.576 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:24.577 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:24.577 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-133619-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:24.577 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-133619-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:24.577 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:24.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:24.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]': finished 2023-07-12T15:31:24.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: osdmap e172: 8 total, 8 up, 8 in 2023-07-12T15:31:24.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2701312756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-133627-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:24.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:24.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-133627-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:24.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:24.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:24.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:24.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:24.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-133619-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:24.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-133619-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:24.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:24.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3469413601' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:24.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:24.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]: dispatch 2023-07-12T15:31:24.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi069-133619-29"}]': finished 2023-07-12T15:31:24.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: osdmap e172: 8 total, 8 up, 8 in 2023-07-12T15:31:24.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2701312756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-133627-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:24.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:24.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-133627-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:24.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:24.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:24.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:24.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:24.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-133619-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:24.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-133619-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:24.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:24.851 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:31:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:31:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:25 smithi145 ceph-mon[95801]: pgmap v202: 292 pgs: 64 unknown, 228 active+clean; 585 KiB data, 356 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:25 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-17","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-133627-30","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-133619-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:25 smithi145 ceph-mon[95801]: osdmap e173: 8 total, 8 up, 8 in 2023-07-12T15:31:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-133619-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-133619-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-17", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:25.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[102777]: pgmap v202: 292 pgs: 64 unknown, 228 active+clean; 585 KiB data, 356 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:25.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:25.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-17","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:25.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-133627-30","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:25.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-133619-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:25.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[102777]: osdmap e173: 8 total, 8 up, 8 in 2023-07-12T15:31:25.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-133619-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:25.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-133619-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:25.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-17", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:25.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:25.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[94241]: pgmap v202: 292 pgs: 64 unknown, 228 active+clean; 585 KiB data, 356 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:25.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:25.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-17","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:25.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi069-133627-30","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:25.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi069-133619-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:25.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[94241]: osdmap e173: 8 total, 8 up, 8 in 2023-07-12T15:31:25.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-133619-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:25.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-133619-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:25.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-17", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:25.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:26.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:31:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:31:26] "GET /metrics HTTP/1.1" 200 235473 "" "Prometheus/2.43.0" 2023-07-12T15:31:27.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:27.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:26 smithi145 ceph-mon[95801]: pgmap v205: 260 pgs: 32 creating+peering, 228 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:27.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-17", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:27.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:26 smithi145 ceph-mon[95801]: osdmap e174: 8 total, 8 up, 8 in 2023-07-12T15:31:27.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-17"}]: dispatch 2023-07-12T15:31:27.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:27.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:26 smithi069 ceph-mon[94241]: pgmap v205: 260 pgs: 32 creating+peering, 228 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:27.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-17", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:27.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:26 smithi069 ceph-mon[94241]: osdmap e174: 8 total, 8 up, 8 in 2023-07-12T15:31:27.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-17"}]: dispatch 2023-07-12T15:31:27.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:27.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:26 smithi069 ceph-mon[102777]: pgmap v205: 260 pgs: 32 creating+peering, 228 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:27.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-17", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:27.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:26 smithi069 ceph-mon[102777]: osdmap e174: 8 total, 8 up, 8 in 2023-07-12T15:31:27.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-17"}]: dispatch 2023-07-12T15:31:28.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:27 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-133619-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-133619-30"}]': finished 2023-07-12T15:31:28.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-17"}]': finished 2023-07-12T15:31:28.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:27 smithi145 ceph-mon[95801]: osdmap e175: 8 total, 8 up, 8 in 2023-07-12T15:31:28.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-17", "mode": "writeback"}]: dispatch 2023-07-12T15:31:28.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3100839750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-133627-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:28.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:27 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-133627-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:28.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:28.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-133619-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-133619-30"}]': finished 2023-07-12T15:31:28.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-17"}]': finished 2023-07-12T15:31:28.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[102777]: osdmap e175: 8 total, 8 up, 8 in 2023-07-12T15:31:28.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-17", "mode": "writeback"}]: dispatch 2023-07-12T15:31:28.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3100839750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-133627-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:28.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-133627-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:28.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:28.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi069-133619-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi069-133619-30"}]': finished 2023-07-12T15:31:28.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-17"}]': finished 2023-07-12T15:31:28.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[94241]: osdmap e175: 8 total, 8 up, 8 in 2023-07-12T15:31:28.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-17", "mode": "writeback"}]: dispatch 2023-07-12T15:31:28.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3100839750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-133627-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:28.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-133627-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:28.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:29.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:28 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:29.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-17", "mode": "writeback"}]': finished 2023-07-12T15:31:29.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:28 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-133627-31","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:29.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:28 smithi145 ceph-mon[95801]: osdmap e176: 8 total, 8 up, 8 in 2023-07-12T15:31:29.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:29.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:29.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:28 smithi145 ceph-mon[95801]: pgmap v208: 300 pgs: 40 unknown, 32 creating+peering, 228 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:29.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:28 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:29.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:28 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:31:29.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:28 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:29.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:29.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-17", "mode": "writeback"}]': finished 2023-07-12T15:31:29.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-133627-31","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:29.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[94241]: osdmap e176: 8 total, 8 up, 8 in 2023-07-12T15:31:29.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:29.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:29.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[94241]: pgmap v208: 300 pgs: 40 unknown, 32 creating+peering, 228 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:29.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:29.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:31:29.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:29.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:29.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-17", "mode": "writeback"}]': finished 2023-07-12T15:31:29.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi069-133627-31","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:29.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[102777]: osdmap e176: 8 total, 8 up, 8 in 2023-07-12T15:31:29.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:29.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:29.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[102777]: pgmap v208: 300 pgs: 40 unknown, 32 creating+peering, 228 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:29.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:29.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:31:29.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:28 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:30.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:30.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:29 smithi145 ceph-mon[95801]: osdmap e177: 8 total, 8 up, 8 in 2023-07-12T15:31:30.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-17"}]: dispatch 2023-07-12T15:31:30.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:30.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:29 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:30.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:30.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:30.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:29 smithi069 ceph-mon[102777]: osdmap e177: 8 total, 8 up, 8 in 2023-07-12T15:31:30.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-17"}]: dispatch 2023-07-12T15:31:30.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:30.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:29 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:30.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:30.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:30.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:29 smithi069 ceph-mon[94241]: osdmap e177: 8 total, 8 up, 8 in 2023-07-12T15:31:30.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-17"}]: dispatch 2023-07-12T15:31:30.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:30.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:29 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:30.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:31.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:31 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:31.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-17"}]': finished 2023-07-12T15:31:31.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:31 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]': finished 2023-07-12T15:31:31.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:31 smithi145 ceph-mon[95801]: osdmap e178: 8 total, 8 up, 8 in 2023-07-12T15:31:31.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:31.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3769990227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-133627-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:31.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:31.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:31 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:31.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:31 smithi145 ceph-mon[95801]: pgmap v211: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2023-07-12T15:31:31.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3769990227' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-133627-32","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:31.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:31 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]': finished 2023-07-12T15:31:31.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:31 smithi145 ceph-mon[95801]: osdmap e179: 8 total, 8 up, 8 in 2023-07-12T15:31:31.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:31.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-17"}]': finished 2023-07-12T15:31:31.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]': finished 2023-07-12T15:31:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[94241]: osdmap e178: 8 total, 8 up, 8 in 2023-07-12T15:31:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3769990227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-133627-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[94241]: pgmap v211: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2023-07-12T15:31:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3769990227' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-133627-32","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:31.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]': finished 2023-07-12T15:31:31.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[94241]: osdmap e179: 8 total, 8 up, 8 in 2023-07-12T15:31:31.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:31.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-17"}]': finished 2023-07-12T15:31:31.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi069-133619-30"}]': finished 2023-07-12T15:31:31.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[102777]: osdmap e178: 8 total, 8 up, 8 in 2023-07-12T15:31:31.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1944258138' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:31.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3769990227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-133627-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:31.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:31.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]: dispatch 2023-07-12T15:31:31.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[102777]: pgmap v211: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2023-07-12T15:31:31.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3769990227' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi069-133627-32","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:31.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi069-133619-30"}]': finished 2023-07-12T15:31:31.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:31 smithi069 ceph-mon[102777]: osdmap e179: 8 total, 8 up, 8 in 2023-07-12T15:31:32.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:32.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:32 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:32 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-133619-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:32.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:32 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-133619-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:32.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:32 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-133619-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:32.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:32 smithi145 ceph-mon[95801]: osdmap e180: 8 total, 8 up, 8 in 2023-07-12T15:31:32.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:32.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-133619-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:32 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-133619-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:32.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:32.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-133619-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:32.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-133619-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:32.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-133619-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:32.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[102777]: osdmap e180: 8 total, 8 up, 8 in 2023-07-12T15:31:32.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:32.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-133619-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-133619-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:32.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:32.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-133619-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:32.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-133619-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:32.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi069-133619-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:32.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[94241]: osdmap e180: 8 total, 8 up, 8 in 2023-07-12T15:31:32.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:32.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-133619-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.494 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-133619-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:32.494 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:33.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:33 smithi145 ceph-mon[95801]: pgmap v214: 260 pgs: 32 unknown, 228 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:31:33.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-19","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:33.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:33.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:33 smithi145 ceph-mon[95801]: osdmap e181: 8 total, 8 up, 8 in 2023-07-12T15:31:33.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/281007377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-133627-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:33.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-19", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:33.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:33 smithi069 ceph-mon[102777]: pgmap v214: 260 pgs: 32 unknown, 228 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:31:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-19","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:33 smithi069 ceph-mon[102777]: osdmap e181: 8 total, 8 up, 8 in 2023-07-12T15:31:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/281007377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-133627-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-19", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:33 smithi069 ceph-mon[94241]: pgmap v214: 260 pgs: 32 unknown, 228 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:31:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-19","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:33 smithi069 ceph-mon[94241]: osdmap e181: 8 total, 8 up, 8 in 2023-07-12T15:31:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/281007377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-133627-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-19", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:34.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:31:34.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-133619-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-133619-31"}]': finished 2023-07-12T15:31:34.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/281007377' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-133627-33","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:34.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-19", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:34.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: osdmap e182: 8 total, 8 up, 8 in 2023-07-12T15:31:34.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-19"}]: dispatch 2023-07-12T15:31:34.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:34.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: pgmap v217: 300 pgs: 72 unknown, 228 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:34.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/281007377' entity='client.admin' cmd=[{ 2023-07-12T15:31:34.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: "prefix": "osd pool set", 2023-07-12T15:31:34.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: "pool": "PoolEIOFlag_smithi069-133627-33", 2023-07-12T15:31:34.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: "var": "eio", 2023-07-12T15:31:34.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: "val": "true" 2023-07-12T15:31:34.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: }]: dispatch 2023-07-12T15:31:34.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:34 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:35.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-133619-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-133619-31"}]': finished 2023-07-12T15:31:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/281007377' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-133627-33","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-19", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: osdmap e182: 8 total, 8 up, 8 in 2023-07-12T15:31:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-19"}]: dispatch 2023-07-12T15:31:35.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:35.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: pgmap v217: 300 pgs: 72 unknown, 228 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:35.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/281007377' entity='client.admin' cmd=[{ 2023-07-12T15:31:35.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: "prefix": "osd pool set", 2023-07-12T15:31:35.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: "pool": "PoolEIOFlag_smithi069-133627-33", 2023-07-12T15:31:35.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: "var": "eio", 2023-07-12T15:31:35.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: "val": "true" 2023-07-12T15:31:35.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: }]: dispatch 2023-07-12T15:31:35.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:35.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi069-133619-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi069-133619-31"}]': finished 2023-07-12T15:31:35.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/281007377' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi069-133627-33","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:35.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-19", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:35.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: osdmap e182: 8 total, 8 up, 8 in 2023-07-12T15:31:35.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-19"}]: dispatch 2023-07-12T15:31:35.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:35.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: pgmap v217: 300 pgs: 72 unknown, 228 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:35.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/281007377' entity='client.admin' cmd=[{ 2023-07-12T15:31:35.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: "prefix": "osd pool set", 2023-07-12T15:31:35.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: "pool": "PoolEIOFlag_smithi069-133627-33", 2023-07-12T15:31:35.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: "var": "eio", 2023-07-12T15:31:35.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: "val": "true" 2023-07-12T15:31:35.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: }]: dispatch 2023-07-12T15:31:35.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:34 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-19"}]': finished 2023-07-12T15:31:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/281007377' entity='client.admin' cmd='[{ 2023-07-12T15:31:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:35 smithi145 ceph-mon[95801]: "prefix": "osd pool set", 2023-07-12T15:31:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:35 smithi145 ceph-mon[95801]: "pool": "PoolEIOFlag_smithi069-133627-33", 2023-07-12T15:31:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:35 smithi145 ceph-mon[95801]: "var": "eio", 2023-07-12T15:31:36.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:35 smithi145 ceph-mon[95801]: "val": "true" 2023-07-12T15:31:36.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:35 smithi145 ceph-mon[95801]: }]': finished 2023-07-12T15:31:36.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:35 smithi145 ceph-mon[95801]: osdmap e183: 8 total, 8 up, 8 in 2023-07-12T15:31:36.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-19", "mode": "writeback"}]: dispatch 2023-07-12T15:31:36.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:36.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-19"}]': finished 2023-07-12T15:31:36.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/281007377' entity='client.admin' cmd='[{ 2023-07-12T15:31:36.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[94241]: "prefix": "osd pool set", 2023-07-12T15:31:36.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[94241]: "pool": "PoolEIOFlag_smithi069-133627-33", 2023-07-12T15:31:36.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[94241]: "var": "eio", 2023-07-12T15:31:36.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[94241]: "val": "true" 2023-07-12T15:31:36.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[94241]: }]': finished 2023-07-12T15:31:36.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[94241]: osdmap e183: 8 total, 8 up, 8 in 2023-07-12T15:31:36.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-19", "mode": "writeback"}]: dispatch 2023-07-12T15:31:36.152 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:36.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-19"}]': finished 2023-07-12T15:31:36.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/281007377' entity='client.admin' cmd='[{ 2023-07-12T15:31:36.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[102777]: "prefix": "osd pool set", 2023-07-12T15:31:36.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[102777]: "pool": "PoolEIOFlag_smithi069-133627-33", 2023-07-12T15:31:36.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[102777]: "var": "eio", 2023-07-12T15:31:36.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[102777]: "val": "true" 2023-07-12T15:31:36.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[102777]: }]': finished 2023-07-12T15:31:36.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[102777]: osdmap e183: 8 total, 8 up, 8 in 2023-07-12T15:31:36.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-19", "mode": "writeback"}]: dispatch 2023-07-12T15:31:36.155 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:36.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:31:36] "GET /metrics HTTP/1.1" 200 235427 "" "Prometheus/2.43.0" 2023-07-12T15:31:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:36 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-19", "mode": "writeback"}]': finished 2023-07-12T15:31:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:36 smithi145 ceph-mon[95801]: osdmap e184: 8 total, 8 up, 8 in 2023-07-12T15:31:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:37.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:36 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:37.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:37.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:36 smithi145 ceph-mon[95801]: pgmap v220: 260 pgs: 260 active+clean; 585 KiB data, 405 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:37.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:36 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2023-07-12T15:31:37.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:37.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:36 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2023-07-12T15:31:37.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:37.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-19", "mode": "writeback"}]': finished 2023-07-12T15:31:37.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[102777]: osdmap e184: 8 total, 8 up, 8 in 2023-07-12T15:31:37.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:37.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:37.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:37.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[102777]: pgmap v220: 260 pgs: 260 active+clean; 585 KiB data, 405 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:37.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2023-07-12T15:31:37.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:37.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2023-07-12T15:31:37.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:37.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-19", "mode": "writeback"}]': finished 2023-07-12T15:31:37.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[94241]: osdmap e184: 8 total, 8 up, 8 in 2023-07-12T15:31:37.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:37.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:37.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:37.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[94241]: pgmap v220: 260 pgs: 260 active+clean; 585 KiB data, 405 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:37.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2023-07-12T15:31:37.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:37.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:36 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2023-07-12T15:31:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:37 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]': finished 2023-07-12T15:31:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:37 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "31"}]': finished 2023-07-12T15:31:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:37 smithi145 ceph-mon[95801]: osdmap e185: 8 total, 8 up, 8 in 2023-07-12T15:31:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3749826962' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-133627-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-19"}]: dispatch 2023-07-12T15:31:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:37 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:37 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-133627-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]: dispatch 2023-07-12T15:31:38.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:37 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]: dispatch 2023-07-12T15:31:38.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]': finished 2023-07-12T15:31:38.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:38.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "31"}]': finished 2023-07-12T15:31:38.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[94241]: osdmap e185: 8 total, 8 up, 8 in 2023-07-12T15:31:38.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:38.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3749826962' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-133627-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:38.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-19"}]: dispatch 2023-07-12T15:31:38.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:38.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-133627-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:38.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:38.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:38.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]: dispatch 2023-07-12T15:31:38.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]: dispatch 2023-07-12T15:31:38.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi069-133619-31"}]': finished 2023-07-12T15:31:38.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:38.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "31"}]': finished 2023-07-12T15:31:38.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[102777]: osdmap e185: 8 total, 8 up, 8 in 2023-07-12T15:31:38.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1331959516' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:38.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3749826962' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-133627-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:38.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-19"}]: dispatch 2023-07-12T15:31:38.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]: dispatch 2023-07-12T15:31:38.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-133627-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:38.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:38.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:38.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]: dispatch 2023-07-12T15:31:38.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:37 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]: dispatch 2023-07-12T15:31:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-19"}]': finished 2023-07-12T15:31:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]': finished 2023-07-12T15:31:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-133627-34","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:39.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]': finished 2023-07-12T15:31:39.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: osdmap e186: 8 total, 8 up, 8 in 2023-07-12T15:31:39.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:39.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:39.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:39.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:39.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:39.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-133619-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:39.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-133619-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:39.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:39.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]: dispatch 2023-07-12T15:31:39.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]: dispatch 2023-07-12T15:31:39.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:38 smithi145 ceph-mon[95801]: pgmap v223: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 405 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:39.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-19"}]': finished 2023-07-12T15:31:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]': finished 2023-07-12T15:31:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-133627-34","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]': finished 2023-07-12T15:31:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: osdmap e186: 8 total, 8 up, 8 in 2023-07-12T15:31:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-133619-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-133619-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:39.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:39.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]: dispatch 2023-07-12T15:31:39.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]: dispatch 2023-07-12T15:31:39.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[102777]: pgmap v223: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 405 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:39.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:39.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-19"}]': finished 2023-07-12T15:31:39.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi069-133619-31"}]': finished 2023-07-12T15:31:39.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi069-133627-34","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:39.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]': finished 2023-07-12T15:31:39.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: osdmap e186: 8 total, 8 up, 8 in 2023-07-12T15:31:39.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:39.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:39.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:39.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:39.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:39.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-133619-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:39.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-133619-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:39.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:39.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]: dispatch 2023-07-12T15:31:39.243 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]: dispatch 2023-07-12T15:31:39.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:38 smithi069 ceph-mon[94241]: pgmap v223: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 405 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:31:40.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:40 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-133619-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:40 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]': finished 2023-07-12T15:31:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:40 smithi145 ceph-mon[95801]: osdmap e187: 8 total, 8 up, 8 in 2023-07-12T15:31:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-133619-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:40 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-133619-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:40.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:40 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-133619-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:40 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]': finished 2023-07-12T15:31:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:40 smithi069 ceph-mon[102777]: osdmap e187: 8 total, 8 up, 8 in 2023-07-12T15:31:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-133619-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:40 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-133619-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:40 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi069-133619-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:40 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pg_num","val":"11"}]': finished 2023-07-12T15:31:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:40 smithi069 ceph-mon[94241]: osdmap e187: 8 total, 8 up, 8 in 2023-07-12T15:31:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-133619-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:40 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-133619-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: pgmap v226: 292 pgs: 64 unknown, 228 active+clean; 585 KiB data, 414 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2023-07-12T15:31:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2023-07-12T15:31:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: osdmap e188: 8 total, 8 up, 8 in 2023-07-12T15:31:41.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2280850796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-133627-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:41.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:41.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2023-07-12T15:31:41.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-133627-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:41.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-133619-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-133619-32"}]': finished 2023-07-12T15:31:41.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-21","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:41.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pg_num_actual", "val": "31"}]': finished 2023-07-12T15:31:41.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-133627-35","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:41.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: osdmap e189: 8 total, 8 up, 8 in 2023-07-12T15:31:41.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:41.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-21", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: pgmap v226: 292 pgs: 64 unknown, 228 active+clean; 585 KiB data, 414 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2023-07-12T15:31:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2023-07-12T15:31:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: osdmap e188: 8 total, 8 up, 8 in 2023-07-12T15:31:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2280850796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-133627-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:41.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:41.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2023-07-12T15:31:41.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-133627-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:41.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-133619-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-133619-32"}]': finished 2023-07-12T15:31:41.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-21","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:41.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pg_num_actual", "val": "31"}]': finished 2023-07-12T15:31:41.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-133627-35","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:41.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: osdmap e189: 8 total, 8 up, 8 in 2023-07-12T15:31:41.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:41.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-21", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:41.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:41.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: pgmap v226: 292 pgs: 64 unknown, 228 active+clean; 585 KiB data, 414 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2023-07-12T15:31:41.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2023-07-12T15:31:41.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: osdmap e188: 8 total, 8 up, 8 in 2023-07-12T15:31:41.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2280850796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-133627-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:41.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:41.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2023-07-12T15:31:41.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-133627-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:41.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi069-133619-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi069-133619-32"}]': finished 2023-07-12T15:31:41.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-21","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:41.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pg_num_actual", "val": "31"}]': finished 2023-07-12T15:31:41.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi069-133627-35","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:41.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: osdmap e189: 8 total, 8 up, 8 in 2023-07-12T15:31:41.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:41.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-21", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-21", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:42 smithi145 ceph-mon[95801]: osdmap e190: 8 total, 8 up, 8 in 2023-07-12T15:31:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:43.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-21"}]: dispatch 2023-07-12T15:31:43.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:42 smithi145 ceph-mon[95801]: pgmap v229: 268 pgs: 40 unknown, 228 active+clean; 585 KiB data, 414 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2023-07-12T15:31:43.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-21", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:42 smithi069 ceph-mon[94241]: osdmap e190: 8 total, 8 up, 8 in 2023-07-12T15:31:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-21"}]: dispatch 2023-07-12T15:31:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:42 smithi069 ceph-mon[94241]: pgmap v229: 268 pgs: 40 unknown, 228 active+clean; 585 KiB data, 414 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2023-07-12T15:31:43.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-21", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:43.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:42 smithi069 ceph-mon[102777]: osdmap e190: 8 total, 8 up, 8 in 2023-07-12T15:31:43.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:43.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-21"}]: dispatch 2023-07-12T15:31:43.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:42 smithi069 ceph-mon[102777]: pgmap v229: 268 pgs: 40 unknown, 228 active+clean; 585 KiB data, 414 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2023-07-12T15:31:44.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-21"}]': finished 2023-07-12T15:31:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:43 smithi145 ceph-mon[95801]: osdmap e191: 8 total, 8 up, 8 in 2023-07-12T15:31:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-21", "mode": "writeback"}]: dispatch 2023-07-12T15:31:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1655837954' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:43 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:44.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:44.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:43 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:44.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:43 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:44.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:43 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:31:44.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-21"}]': finished 2023-07-12T15:31:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[102777]: osdmap e191: 8 total, 8 up, 8 in 2023-07-12T15:31:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-21", "mode": "writeback"}]: dispatch 2023-07-12T15:31:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1655837954' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:44.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:44.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:44.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:31:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-21"}]': finished 2023-07-12T15:31:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[94241]: osdmap e191: 8 total, 8 up, 8 in 2023-07-12T15:31:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-21", "mode": "writeback"}]: dispatch 2023-07-12T15:31:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1655837954' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:31:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:43 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:31:44.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:31:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:31:45.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-21", "mode": "writeback"}]': finished 2023-07-12T15:31:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-36","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]': finished 2023-07-12T15:31:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[94241]: osdmap e192: 8 total, 8 up, 8 in 2023-07-12T15:31:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[94241]: pgmap v232: 291 pgs: 64 unknown, 227 active+clean; 585 KiB data, 414 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:45.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:45.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-21", "mode": "writeback"}]': finished 2023-07-12T15:31:45.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-36","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:45.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]': finished 2023-07-12T15:31:45.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[102777]: osdmap e192: 8 total, 8 up, 8 in 2023-07-12T15:31:45.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:45.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:45.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:45.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[102777]: pgmap v232: 291 pgs: 64 unknown, 227 active+clean; 585 KiB data, 414 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:45.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:45.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:45.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:44 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-21", "mode": "writeback"}]': finished 2023-07-12T15:31:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:44 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-36","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:44 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi069-133619-32"}]': finished 2023-07-12T15:31:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:44 smithi145 ceph-mon[95801]: osdmap e192: 8 total, 8 up, 8 in 2023-07-12T15:31:45.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4046206723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:45.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:45.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:44 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]: dispatch 2023-07-12T15:31:45.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:44 smithi145 ceph-mon[95801]: pgmap v232: 291 pgs: 64 unknown, 227 active+clean; 585 KiB data, 414 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:45.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:45.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:44 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:45.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:46.152 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]': finished 2023-07-12T15:31:46.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:46.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[102777]: osdmap e193: 8 total, 8 up, 8 in 2023-07-12T15:31:46.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2423171630' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:46.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-21"}]: dispatch 2023-07-12T15:31:46.153 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:46.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:46.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-133619-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:46.154 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:46.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]': finished 2023-07-12T15:31:46.154 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:46.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[94241]: osdmap e193: 8 total, 8 up, 8 in 2023-07-12T15:31:46.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2423171630' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:46.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-21"}]: dispatch 2023-07-12T15:31:46.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:46.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:46.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-133619-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:46.156 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:45 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi069-133619-32"}]': finished 2023-07-12T15:31:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:45 smithi145 ceph-mon[95801]: osdmap e193: 8 total, 8 up, 8 in 2023-07-12T15:31:46.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2423171630' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:46.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-21"}]: dispatch 2023-07-12T15:31:46.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:46.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:46.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-133619-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:46.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:46.488 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:31:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:31:46] "GET /metrics HTTP/1.1" 200 233018 "" "Prometheus/2.43.0" 2023-07-12T15:31:47.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:47 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2423171630' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-37","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-21"}]': finished 2023-07-12T15:31:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-133619-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:47 smithi145 ceph-mon[95801]: osdmap e194: 8 total, 8 up, 8 in 2023-07-12T15:31:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-133619-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:47.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:47 smithi145 ceph-mon[95801]: pgmap v235: 323 pgs: 32 unknown, 291 active+clean; 586 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2023-07-12T15:31:47.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:47.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:47.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2423171630' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-37","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:47.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-21"}]': finished 2023-07-12T15:31:47.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-133619-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:47.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[102777]: osdmap e194: 8 total, 8 up, 8 in 2023-07-12T15:31:47.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-133619-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:47.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[102777]: pgmap v235: 323 pgs: 32 unknown, 291 active+clean; 586 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2023-07-12T15:31:47.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:47.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:47.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2423171630' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-37","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:47.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-21"}]': finished 2023-07-12T15:31:47.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi069-133619-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:47.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[94241]: osdmap e194: 8 total, 8 up, 8 in 2023-07-12T15:31:47.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-133619-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:47.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[94241]: pgmap v235: 323 pgs: 32 unknown, 291 active+clean; 586 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2023-07-12T15:31:47.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:48.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:48 smithi145 ceph-mon[95801]: osdmap e195: 8 total, 8 up, 8 in 2023-07-12T15:31:48.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/966592340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:48.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:48.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:48.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-133619-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-133619-33"}]': finished 2023-07-12T15:31:48.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-38","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:48.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:48 smithi145 ceph-mon[95801]: osdmap e196: 8 total, 8 up, 8 in 2023-07-12T15:31:48.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:48.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[102777]: osdmap e195: 8 total, 8 up, 8 in 2023-07-12T15:31:48.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/966592340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:48.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:48.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:48.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-133619-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-133619-33"}]': finished 2023-07-12T15:31:48.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-38","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:48.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[102777]: osdmap e196: 8 total, 8 up, 8 in 2023-07-12T15:31:48.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:48.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[94241]: osdmap e195: 8 total, 8 up, 8 in 2023-07-12T15:31:48.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/966592340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:48.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:48.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:48.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi069-133619-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi069-133619-33"}]': finished 2023-07-12T15:31:48.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi069-133627-38","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:48.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[94241]: osdmap e196: 8 total, 8 up, 8 in 2023-07-12T15:31:48.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:49 smithi145 ceph-mon[95801]: pgmap v238: 363 pgs: 104 unknown, 259 active+clean; 586 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2023-07-12T15:31:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-23","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:49 smithi145 ceph-mon[95801]: osdmap e197: 8 total, 8 up, 8 in 2023-07-12T15:31:49.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:49 smithi069 ceph-mon[94241]: pgmap v238: 363 pgs: 104 unknown, 259 active+clean; 586 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2023-07-12T15:31:49.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-23","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:49 smithi069 ceph-mon[94241]: osdmap e197: 8 total, 8 up, 8 in 2023-07-12T15:31:49.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:49 smithi069 ceph-mon[102777]: pgmap v238: 363 pgs: 104 unknown, 259 active+clean; 586 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2023-07-12T15:31:49.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:49.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-23","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:49.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:49 smithi069 ceph-mon[102777]: osdmap e197: 8 total, 8 up, 8 in 2023-07-12T15:31:50.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:50.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:50 smithi069 ceph-mon[102777]: osdmap e198: 8 total, 8 up, 8 in 2023-07-12T15:31:50.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:50.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-23", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:50.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:50 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2023-07-12T15:31:50.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:50 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2023-07-12T15:31:50.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:50.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:50 smithi069 ceph-mon[94241]: osdmap e198: 8 total, 8 up, 8 in 2023-07-12T15:31:50.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:50.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-23", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:50.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:50 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2023-07-12T15:31:50.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:50 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2023-07-12T15:31:50.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:50.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:50 smithi145 ceph-mon[95801]: osdmap e198: 8 total, 8 up, 8 in 2023-07-12T15:31:50.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:50.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-23", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:31:50.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:50 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2023-07-12T15:31:50.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:50 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2023-07-12T15:31:51.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[94241]: pgmap v241: 291 pgs: 291 active+clean; 586 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2023-07-12T15:31:51.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:51.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-133619-33"}]': finished 2023-07-12T15:31:51.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-23", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:51.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "30"}]': finished 2023-07-12T15:31:51.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[94241]: osdmap e199: 8 total, 8 up, 8 in 2023-07-12T15:31:51.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-23"}]: dispatch 2023-07-12T15:31:51.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:51.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[102777]: pgmap v241: 291 pgs: 291 active+clean; 586 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2023-07-12T15:31:51.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:51.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-133619-33"}]': finished 2023-07-12T15:31:51.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-23", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:51.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "30"}]': finished 2023-07-12T15:31:51.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[102777]: osdmap e199: 8 total, 8 up, 8 in 2023-07-12T15:31:51.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-23"}]: dispatch 2023-07-12T15:31:51.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:51.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:51 smithi145 ceph-mon[95801]: pgmap v241: 291 pgs: 291 active+clean; 586 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2023-07-12T15:31:51.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:51.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi069-133619-33"}]': finished 2023-07-12T15:31:51.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-23", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:31:51.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:51 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi069-133656-1", "var": "pgp_num_actual", "val": "30"}]': finished 2023-07-12T15:31:51.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:51 smithi145 ceph-mon[95801]: osdmap e199: 8 total, 8 up, 8 in 2023-07-12T15:31:51.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-23"}]: dispatch 2023-07-12T15:31:51.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-133619-33"}]: dispatch 2023-07-12T15:31:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]: dispatch 2023-07-12T15:31:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]: dispatch 2023-07-12T15:31:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-23"}]': finished 2023-07-12T15:31:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-133619-33"}]': finished 2023-07-12T15:31:53.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]': finished 2023-07-12T15:31:53.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: osdmap e200: 8 total, 8 up, 8 in 2023-07-12T15:31:53.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-23", "mode": "writeback"}]: dispatch 2023-07-12T15:31:53.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/455129643' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:53.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:53.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:53.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:53.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:53.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi069-133619-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:53.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:52 smithi145 ceph-mon[95801]: pgmap v244: 291 pgs: 32 unknown, 259 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2023-07-12T15:31:53.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]: dispatch 2023-07-12T15:31:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]: dispatch 2023-07-12T15:31:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-23"}]': finished 2023-07-12T15:31:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-133619-33"}]': finished 2023-07-12T15:31:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]': finished 2023-07-12T15:31:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: osdmap e200: 8 total, 8 up, 8 in 2023-07-12T15:31:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-23", "mode": "writeback"}]: dispatch 2023-07-12T15:31:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/455129643' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:53.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi069-133619-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:53.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[94241]: pgmap v244: 291 pgs: 32 unknown, 259 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2023-07-12T15:31:53.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:53.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:53.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]: dispatch 2023-07-12T15:31:53.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]: dispatch 2023-07-12T15:31:53.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-23"}]': finished 2023-07-12T15:31:53.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2482101579' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi069-133619-33"}]': finished 2023-07-12T15:31:53.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi069-133656-1","var":"pgp_num","val":"11"}]': finished 2023-07-12T15:31:53.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: osdmap e200: 8 total, 8 up, 8 in 2023-07-12T15:31:53.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-23", "mode": "writeback"}]: dispatch 2023-07-12T15:31:53.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/455129643' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:53.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1115593566' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2023-07-12T15:31:53.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:53.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:53.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:53.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi069-133619-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:53.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:52 smithi069 ceph-mon[102777]: pgmap v244: 291 pgs: 32 unknown, 259 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2023-07-12T15:31:54.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:53 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:54.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-23", "mode": "writeback"}]': finished 2023-07-12T15:31:54.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-39","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:54.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi069-133619-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:54.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:53 smithi145 ceph-mon[95801]: osdmap e201: 8 total, 8 up, 8 in 2023-07-12T15:31:54.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi069-133619-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:54.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:54.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:54.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:54.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:54.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-133656-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:54.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-133656-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-23", "mode": "writeback"}]': finished 2023-07-12T15:31:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-39","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi069-133619-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[102777]: osdmap e201: 8 total, 8 up, 8 in 2023-07-12T15:31:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi069-133619-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-133656-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-133656-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:54.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:31:54.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-23", "mode": "writeback"}]': finished 2023-07-12T15:31:54.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-39","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:54.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi069-133619-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:54.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[94241]: osdmap e201: 8 total, 8 up, 8 in 2023-07-12T15:31:54.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi069-133619-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:54.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:54.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:54.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:54.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:54.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-133656-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:54.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-133656-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:31:54.892 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:31:54 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:31:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[94241]: pgmap v247: 292 pgs: 64 unknown, 228 active+clean; 583 KiB data, 415 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-133656-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[94241]: osdmap e202: 8 total, 8 up, 8 in 2023-07-12T15:31:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-133656-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3714004361' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-133656-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[102777]: pgmap v247: 292 pgs: 64 unknown, 228 active+clean; 583 KiB data, 415 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-133656-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[102777]: osdmap e202: 8 total, 8 up, 8 in 2023-07-12T15:31:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-133656-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3714004361' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-133656-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:54 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:54 smithi145 ceph-mon[95801]: pgmap v247: 292 pgs: 64 unknown, 228 active+clean; 583 KiB data, 415 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:31:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi069-133656-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:31:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:54 smithi145 ceph-mon[95801]: osdmap e202: 8 total, 8 up, 8 in 2023-07-12T15:31:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-133656-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3714004361' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-133656-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:31:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:54 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:31:56.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi069-133619-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi069-133619-34"}]': finished 2023-07-12T15:31:56.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3714004361' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-40","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:56.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:56.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:55 smithi069 ceph-mon[102777]: osdmap e203: 8 total, 8 up, 8 in 2023-07-12T15:31:56.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-23"}]: dispatch 2023-07-12T15:31:56.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi069-133619-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi069-133619-34"}]': finished 2023-07-12T15:31:56.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3714004361' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-40","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:56.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:56.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:55 smithi069 ceph-mon[94241]: osdmap e203: 8 total, 8 up, 8 in 2023-07-12T15:31:56.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-23"}]: dispatch 2023-07-12T15:31:56.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi069-133619-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi069-133619-34"}]': finished 2023-07-12T15:31:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3714004361' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-40","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:31:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:55 smithi145 ceph-mon[95801]: osdmap e203: 8 total, 8 up, 8 in 2023-07-12T15:31:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-23"}]: dispatch 2023-07-12T15:31:56.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:31:56] "GET /metrics HTTP/1.1" 200 233018 "" "Prometheus/2.43.0" 2023-07-12T15:31:57.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:6cac518f:::0:head expected=9:6cac518f:::0:head 2023-07-12T15:31:57.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:6cac518f:::0:head -> 0 2023-07-12T15:31:57.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=0 expected=0 2023-07-12T15:31:57.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:62a1935d:::14:head 2023-07-12T15:31:57.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:62a1935d:::14:head expected=9:62a1935d:::14:head 2023-07-12T15:31:57.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:62a1935d:::14:head -> 14 2023-07-12T15:31:57.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=14 expected=14 2023-07-12T15:31:57.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2023-07-12T15:31:57.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:5c6b0b28:::7:head expected=9:5c6b0b28:::7:head 2023-07-12T15:31:57.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2023-07-12T15:31:57.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=7 expected=7 2023-07-12T15:31:57.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:cfc208b3:::3:head 2023-07-12T15:31:57.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:cfc208b3:::3:head expected=9:cfc208b3:::3:head 2023-07-12T15:31:57.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2023-07-12T15:31:57.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=3 expected=3 2023-07-12T15:31:57.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2023-07-12T15:31:57.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:c4fdafeb:::6:head expected=9:c4fdafeb:::6:head 2023-07-12T15:31:57.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2023-07-12T15:31:57.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=6 expected=6 2023-07-12T15:31:57.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2023-07-12T15:31:57.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:bd63b0f1:::8:head expected=9:bd63b0f1:::8:head 2023-07-12T15:31:57.024 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2023-07-12T15:31:57.025 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=8 expected=8 2023-07-12T15:31:57.025 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:b29083e3:::5:head 2023-07-12T15:31:57.025 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:b29083e3:::5:head expected=9:b29083e3:::5:head 2023-07-12T15:31:57.025 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:b29083e3:::5:head -> 5 2023-07-12T15:31:57.025 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=5 expected=5 2023-07-12T15:31:57.025 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 9:566253c9:::13:head 2023-07-12T15:31:57.025 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=9:566253c9:::13:head expected=9:566253c9:::13:head 2023-07-12T15:31:57.025 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 9:566253c9:::13:head -> 13 2023-07-12T15:31:57.026 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=13 expected=13 2023-07-12T15:31:57.026 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (176 ms) 2023-07-12T15:31:57.026 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-07-12T15:31:57.026 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (110917 ms) 2023-07-12T15:31:57.026 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-07-12T15:31:57.026 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 0/5 -> MIN 9:33333333::::head 2023-07-12T15:31:57.026 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 1/5 -> 9:33333333::::head 9:66666666::::head 2023-07-12T15:31:57.027 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 2/5 -> 9:66666666::::head 9:99999999::::head 2023-07-12T15:31:57.027 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 3/5 -> 9:99999999::::head 9:cccccccc::::head 2023-07-12T15:31:57.027 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 4/5 -> 9:cccccccc::::head MAX 2023-07-12T15:31:57.027 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (14067 ms) 2023-07-12T15:31:57.027 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 7 tests from LibRadosList (125367 ms total) 2023-07-12T15:31:57.027 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2023-07-12T15:31:57.027 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-07-12T15:31:57.028 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-07-12T15:31:57.028 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1058 ms) 2023-07-12T15:31:57.028 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-07-12T15:31:57.028 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo2,foo3 2023-07-12T15:31:57.028 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2023-07-12T15:31:57.028 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo2 2023-07-12T15:31:57.028 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo3 2023-07-12T15:31:57.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo4,foo5 2023-07-12T15:31:57.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo4 2023-07-12T15:31:57.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo5 2023-07-12T15:31:57.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2023-07-12T15:31:57.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo6,foo7 2023-07-12T15:31:57.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo7 2023-07-12T15:31:57.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo6 2023-07-12T15:31:57.029 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-07-12T15:31:57.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo4 2023-07-12T15:31:57.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo5 2023-07-12T15:31:57.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo7 2023-07-12T15:31:57.030 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo6 2023-07-12T15:31:57.031 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo1 2023-07-12T15:31:57.031 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo1 2023-07-12T15:31:57.031 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo2 2023-07-12T15:31:57.032 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo3 2023-07-12T15:31:57.032 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (29 ms) 2023-07-12T15:31:57.032 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2023-07-12T15:31:57.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 1 0 2023-07-12T15:31:57.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 10 0 2023-07-12T15:31:57.033 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 13 0 2023-07-12T15:31:57.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-133656-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-133656-2"}]': finished 2023-07-12T15:31:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-23"}]': finished 2023-07-12T15:31:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[102777]: osdmap e204: 8 total, 8 up, 8 in 2023-07-12T15:31:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3165091993' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[102777]: pgmap v250: 340 pgs: 48 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 424 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2023-07-12T15:31:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-133656-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-133656-2"}]': finished 2023-07-12T15:31:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-23"}]': finished 2023-07-12T15:31:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[94241]: osdmap e204: 8 total, 8 up, 8 in 2023-07-12T15:31:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3165091993' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:57.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:56 smithi069 ceph-mon[94241]: pgmap v250: 340 pgs: 48 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 424 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2023-07-12T15:31:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:56 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:31:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:56 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi069-133656-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi069-133656-2"}]': finished 2023-07-12T15:31:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-23"}]': finished 2023-07-12T15:31:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:56 smithi145 ceph-mon[95801]: osdmap e204: 8 total, 8 up, 8 in 2023-07-12T15:31:57.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3165091993' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:57.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:56 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:57.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:56 smithi145 ceph-mon[95801]: pgmap v250: 340 pgs: 48 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 424 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2023-07-12T15:31:58.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-41","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:58.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:57 smithi069 ceph-mon[102777]: osdmap e205: 8 total, 8 up, 8 in 2023-07-12T15:31:58.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-41","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:57 smithi069 ceph-mon[94241]: osdmap e205: 8 total, 8 up, 8 in 2023-07-12T15:31:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:58.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-41","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:58.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:57 smithi145 ceph-mon[95801]: osdmap e205: 8 total, 8 up, 8 in 2023-07-12T15:31:58.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:59.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-133619-34"}]': finished 2023-07-12T15:31:59.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: osdmap e206: 8 total, 8 up, 8 in 2023-07-12T15:31:59.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1683533281' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:59.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:59.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:59.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:59.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:59.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:59.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: pgmap v253: 356 pgs: 96 unknown, 32 creating+peering, 228 active+clean; 584 KiB data, 424 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2023-07-12T15:31:59.242 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:31:59.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-133619-34"}]': finished 2023-07-12T15:31:59.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-42","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:59.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-25","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:59.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]': finished 2023-07-12T15:31:59.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: osdmap e207: 8 total, 8 up, 8 in 2023-07-12T15:31:59.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:59.243 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:59.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-133619-34"}]': finished 2023-07-12T15:31:59.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: osdmap e206: 8 total, 8 up, 8 in 2023-07-12T15:31:59.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1683533281' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:59.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:59.244 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:59.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:59.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:59.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:59.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: pgmap v253: 356 pgs: 96 unknown, 32 creating+peering, 228 active+clean; 584 KiB data, 424 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2023-07-12T15:31:59.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:31:59.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-133619-34"}]': finished 2023-07-12T15:31:59.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-42","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:59.245 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-25","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:59.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]': finished 2023-07-12T15:31:59.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: osdmap e207: 8 total, 8 up, 8 in 2023-07-12T15:31:59.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:59.246 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:58 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:59.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi069-133619-34"}]': finished 2023-07-12T15:31:59.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: osdmap e206: 8 total, 8 up, 8 in 2023-07-12T15:31:59.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1683533281' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:59.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-133619-34"}]: dispatch 2023-07-12T15:31:59.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:59.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:59.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:31:59.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:59.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: pgmap v253: 356 pgs: 96 unknown, 32 creating+peering, 228 active+clean; 584 KiB data, 424 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2023-07-12T15:31:59.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:31:59.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/997786855' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi069-133619-34"}]': finished 2023-07-12T15:31:59.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-42","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:59.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-25","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:31:59.359 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi069-133656-2"}]': finished 2023-07-12T15:31:59.359 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: osdmap e207: 8 total, 8 up, 8 in 2023-07-12T15:31:59.359 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/621347425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:31:59.359 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:58 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]: dispatch 2023-07-12T15:32:00.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:00.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:00.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi069-133619-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:00.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]': finished 2023-07-12T15:32:00.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi069-133619-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:00.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[102777]: osdmap e208: 8 total, 8 up, 8 in 2023-07-12T15:32:00.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi069-133619-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:00.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2379117471' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:00.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-25", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:00.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:00.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:00.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi069-133619-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:00.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]': finished 2023-07-12T15:32:00.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi069-133619-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:00.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[94241]: osdmap e208: 8 total, 8 up, 8 in 2023-07-12T15:32:00.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi069-133619-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:00.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2379117471' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:00.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:31:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-25", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:00.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:00.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:00.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi069-133619-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:00.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:59 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi069-133656-2"}]': finished 2023-07-12T15:32:00.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi069-133619-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:00.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:59 smithi145 ceph-mon[95801]: osdmap e208: 8 total, 8 up, 8 in 2023-07-12T15:32:00.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi069-133619-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:00.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2379117471' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:00.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:31:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-25", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:01.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:01 smithi069 ceph-mon[94241]: pgmap v256: 388 pgs: 32 unknown, 356 active+clean; 584 KiB data, 446 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:01.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2379117471' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-43","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:01.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:01 smithi069 ceph-mon[102777]: pgmap v256: 388 pgs: 32 unknown, 356 active+clean; 584 KiB data, 446 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:01.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2379117471' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-43","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:01.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:01 smithi145 ceph-mon[95801]: pgmap v256: 388 pgs: 32 unknown, 356 active+clean; 584 KiB data, 446 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:01.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2379117471' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi069-133627-43","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:02.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-25", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[102777]: osdmap e209: 8 total, 8 up, 8 in 2023-07-12T15:32:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-25"}]: dispatch 2023-07-12T15:32:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4210865129' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi069-133619-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi069-133619-35"}]': finished 2023-07-12T15:32:02.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-25"}]': finished 2023-07-12T15:32:02.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133656-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:02.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[102777]: osdmap e210: 8 total, 8 up, 8 in 2023-07-12T15:32:02.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-25", "mode": "writeback"}]: dispatch 2023-07-12T15:32:02.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4210865129' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-133656-3","pool2":"test-rados-api-smithi069-133656-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2023-07-12T15:32:02.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-133656-3","pool2":"test-rados-api-smithi069-133656-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2023-07-12T15:32:02.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-25", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:02.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[94241]: osdmap e209: 8 total, 8 up, 8 in 2023-07-12T15:32:02.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-25"}]: dispatch 2023-07-12T15:32:02.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4210865129' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:02.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:02.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi069-133619-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi069-133619-35"}]': finished 2023-07-12T15:32:02.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-25"}]': finished 2023-07-12T15:32:02.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133656-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:02.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[94241]: osdmap e210: 8 total, 8 up, 8 in 2023-07-12T15:32:02.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-25", "mode": "writeback"}]: dispatch 2023-07-12T15:32:02.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4210865129' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-133656-3","pool2":"test-rados-api-smithi069-133656-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2023-07-12T15:32:02.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:02 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-133656-3","pool2":"test-rados-api-smithi069-133656-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2023-07-12T15:32:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-25", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:02 smithi145 ceph-mon[95801]: osdmap e209: 8 total, 8 up, 8 in 2023-07-12T15:32:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-25"}]: dispatch 2023-07-12T15:32:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4210865129' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:02 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133656-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:02.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi069-133619-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi069-133619-35"}]': finished 2023-07-12T15:32:02.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-25"}]': finished 2023-07-12T15:32:02.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:02 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133656-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:02.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:02 smithi145 ceph-mon[95801]: osdmap e210: 8 total, 8 up, 8 in 2023-07-12T15:32:02.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-25", "mode": "writeback"}]: dispatch 2023-07-12T15:32:02.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4210865129' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-133656-3","pool2":"test-rados-api-smithi069-133656-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2023-07-12T15:32:02.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:02 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-133656-3","pool2":"test-rados-api-smithi069-133656-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2023-07-12T15:32:02.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 7 0 2023-07-12T15:32:02.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 14 0 2023-07-12T15:32:02.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 0 0 2023-07-12T15:32:02.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 15 0 2023-07-12T15:32:02.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 11 0 2023-07-12T15:32:02.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 5 0 2023-07-12T15:32:02.805 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 8 0 2023-07-12T15:32:02.805 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 6 0 2023-07-12T15:32:02.805 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 3 0 2023-07-12T15:32:02.805 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 4 0 2023-07-12T15:32:02.805 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 12 0 2023-07-12T15:32:02.805 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 9 0 2023-07-12T15:32:02.806 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2 0 2023-07-12T15:32:02.806 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-07-12T15:32:02.806 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (44 ms) 2023-07-12T15:32:02.806 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 3 tests from LibRadosListEC (1131 ms total) 2023-07-12T15:32:02.806 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2023-07-12T15:32:02.807 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-07-12T15:32:02.807 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-07-12T15:32:02.807 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2859 ms) 2023-07-12T15:32:02.807 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 1 test from LibRadosListNP (2859 ms total) 2023-07-12T15:32:02.807 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2023-07-12T15:32:02.807 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] Global test environment tear-down 2023-07-12T15:32:02.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [==========] 11 tests from 3 test suites ran. (142126 ms total) 2023-07-12T15:32:02.808 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ PASSED ] 11 tests. 2023-07-12T15:32:02.808 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:32:02.808 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133927 2023-07-12T15:32:02.809 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133927 2023-07-12T15:32:02.809 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:32:02.809 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133901 2023-07-12T15:32:02.809 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133901 2023-07-12T15:32:02.809 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:32:02.809 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134075 2023-07-12T15:32:02.810 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 134075 2023-07-12T15:32:02.810 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:32:02.810 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133622 2023-07-12T15:32:02.810 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133622 2023-07-12T15:32:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:03 smithi069 ceph-mon[102777]: pgmap v259: 396 pgs: 40 unknown, 356 active+clean; 584 KiB data, 446 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:32:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:03 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-25", "mode": "writeback"}]': finished 2023-07-12T15:32:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:03 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-133656-3","pool2":"test-rados-api-smithi069-133656-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2023-07-12T15:32:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:03 smithi069 ceph-mon[102777]: osdmap e211: 8 total, 8 up, 8 in 2023-07-12T15:32:03.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:03 smithi069 ceph-mon[94241]: pgmap v259: 396 pgs: 40 unknown, 356 active+clean; 584 KiB data, 446 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:32:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:03 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-25", "mode": "writeback"}]': finished 2023-07-12T15:32:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:03 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-133656-3","pool2":"test-rados-api-smithi069-133656-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2023-07-12T15:32:03.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:03 smithi069 ceph-mon[94241]: osdmap e211: 8 total, 8 up, 8 in 2023-07-12T15:32:03.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:03 smithi145 ceph-mon[95801]: pgmap v259: 396 pgs: 40 unknown, 356 active+clean; 584 KiB data, 446 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:32:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:03 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-25", "mode": "writeback"}]': finished 2023-07-12T15:32:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:03 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi069-133656-3","pool2":"test-rados-api-smithi069-133656-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2023-07-12T15:32:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:03 smithi145 ceph-mon[95801]: osdmap e211: 8 total, 8 up, 8 in 2023-07-12T15:32:03.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:05.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:05.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:04 smithi145 ceph-mon[95801]: osdmap e212: 8 total, 8 up, 8 in 2023-07-12T15:32:05.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-25"}]: dispatch 2023-07-12T15:32:05.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:05.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:04 smithi145 ceph-mon[95801]: pgmap v262: 292 pgs: 292 active+clean; 584 KiB data, 446 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:05.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:04 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:05.102 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:32:04 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:32:05.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:04 smithi069 ceph-mon[102777]: osdmap e212: 8 total, 8 up, 8 in 2023-07-12T15:32:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-25"}]: dispatch 2023-07-12T15:32:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:04 smithi069 ceph-mon[102777]: pgmap v262: 292 pgs: 292 active+clean; 584 KiB data, 446 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:04 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:04 smithi069 ceph-mon[94241]: osdmap e212: 8 total, 8 up, 8 in 2023-07-12T15:32:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-25"}]: dispatch 2023-07-12T15:32:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:04 smithi069 ceph-mon[94241]: pgmap v262: 292 pgs: 292 active+clean; 584 KiB data, 446 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:04 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:06 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-25"}]': finished 2023-07-12T15:32:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-133619-35"}]': finished 2023-07-12T15:32:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:06 smithi145 ceph-mon[95801]: osdmap e213: 8 total, 8 up, 8 in 2023-07-12T15:32:06.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:06.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:32:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:32:06] "GET /metrics HTTP/1.1" 200 237965 "" "Prometheus/2.43.0" 2023-07-12T15:32:06.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:06 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:06.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-25"}]': finished 2023-07-12T15:32:06.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-133619-35"}]': finished 2023-07-12T15:32:06.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:06 smithi069 ceph-mon[94241]: osdmap e213: 8 total, 8 up, 8 in 2023-07-12T15:32:06.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:06 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-25"}]': finished 2023-07-12T15:32:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi069-133619-35"}]': finished 2023-07-12T15:32:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:06 smithi069 ceph-mon[102777]: osdmap e213: 8 total, 8 up, 8 in 2023-07-12T15:32:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-133619-35"}]: dispatch 2023-07-12T15:32:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-133619-35"}]': finished 2023-07-12T15:32:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: osdmap e214: 8 total, 8 up, 8 in 2023-07-12T15:32:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: pgmap v265: 196 pgs: 196 active+clean; 583 KiB data, 447 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-133619-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:07.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-133619-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:07.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-133619-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:07.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: osdmap e215: 8 total, 8 up, 8 in 2023-07-12T15:32:07.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-133619-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-133619-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3357967527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-133627-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:07.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-133627-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:07.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:07.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-133619-35"}]': finished 2023-07-12T15:32:07.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: osdmap e214: 8 total, 8 up, 8 in 2023-07-12T15:32:07.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: pgmap v265: 196 pgs: 196 active+clean; 583 KiB data, 447 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:07.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-133619-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:07.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-133619-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:07.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-133619-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:07.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: osdmap e215: 8 total, 8 up, 8 in 2023-07-12T15:32:07.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-133619-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-133619-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3357967527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-133627-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:07.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-133627-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:07.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:07.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1524868886' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi069-133619-35"}]': finished 2023-07-12T15:32:07.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: osdmap e214: 8 total, 8 up, 8 in 2023-07-12T15:32:07.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: pgmap v265: 196 pgs: 196 active+clean; 583 KiB data, 447 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:07.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-133619-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:07.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-133619-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:07.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi069-133619-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:07.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: osdmap e215: 8 total, 8 up, 8 in 2023-07-12T15:32:07.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-133619-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-133619-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:07.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3357967527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-133627-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:07.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-133627-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:07.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:08 smithi145 ceph-mon[95801]: pgmap v268: 260 pgs: 64 unknown, 196 active+clean; 583 KiB data, 447 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:08 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-133627-44","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-27","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:08 smithi145 ceph-mon[95801]: osdmap e216: 8 total, 8 up, 8 in 2023-07-12T15:32:09.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:08 smithi069 ceph-mon[102777]: pgmap v268: 260 pgs: 64 unknown, 196 active+clean; 583 KiB data, 447 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:09.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:08 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-133627-44","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:09.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-27","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:09.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:08 smithi069 ceph-mon[102777]: osdmap e216: 8 total, 8 up, 8 in 2023-07-12T15:32:09.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:08 smithi069 ceph-mon[94241]: pgmap v268: 260 pgs: 64 unknown, 196 active+clean; 583 KiB data, 447 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:09.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:08 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi069-133627-44","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:09.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-27","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:09.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:08 smithi069 ceph-mon[94241]: osdmap e216: 8 total, 8 up, 8 in 2023-07-12T15:32:10.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:09 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-133619-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-133619-36"}]': finished 2023-07-12T15:32:10.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:09 smithi145 ceph-mon[95801]: osdmap e217: 8 total, 8 up, 8 in 2023-07-12T15:32:10.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-27", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:10.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:09 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-133619-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-133619-36"}]': finished 2023-07-12T15:32:10.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:09 smithi069 ceph-mon[102777]: osdmap e217: 8 total, 8 up, 8 in 2023-07-12T15:32:10.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-27", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:10.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:09 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi069-133619-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi069-133619-36"}]': finished 2023-07-12T15:32:10.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:09 smithi069 ceph-mon[94241]: osdmap e217: 8 total, 8 up, 8 in 2023-07-12T15:32:10.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-27", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:11.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-27", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:11.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:10 smithi145 ceph-mon[95801]: osdmap e218: 8 total, 8 up, 8 in 2023-07-12T15:32:11.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-27"}]: dispatch 2023-07-12T15:32:11.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/512127389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-133627-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:11.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:10 smithi145 ceph-mon[95801]: pgmap v271: 268 pgs: 32 unknown, 8 creating+peering, 228 active+clean; 583 KiB data, 487 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2023-07-12T15:32:11.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-27", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:10 smithi069 ceph-mon[102777]: osdmap e218: 8 total, 8 up, 8 in 2023-07-12T15:32:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-27"}]: dispatch 2023-07-12T15:32:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/512127389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-133627-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:10 smithi069 ceph-mon[102777]: pgmap v271: 268 pgs: 32 unknown, 8 creating+peering, 228 active+clean; 583 KiB data, 487 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2023-07-12T15:32:11.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-27", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:11.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:10 smithi069 ceph-mon[94241]: osdmap e218: 8 total, 8 up, 8 in 2023-07-12T15:32:11.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-27"}]: dispatch 2023-07-12T15:32:11.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/512127389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-133627-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:11.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:10 smithi069 ceph-mon[94241]: pgmap v271: 268 pgs: 32 unknown, 8 creating+peering, 228 active+clean; 583 KiB data, 487 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2023-07-12T15:32:12.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-27"}]': finished 2023-07-12T15:32:12.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/512127389' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-133627-45","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:12.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:11 smithi145 ceph-mon[95801]: osdmap e219: 8 total, 8 up, 8 in 2023-07-12T15:32:12.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-27", "mode": "writeback"}]: dispatch 2023-07-12T15:32:12.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:12.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:11 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:12.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-27"}]': finished 2023-07-12T15:32:12.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/512127389' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-133627-45","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:12.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:11 smithi069 ceph-mon[102777]: osdmap e219: 8 total, 8 up, 8 in 2023-07-12T15:32:12.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-27", "mode": "writeback"}]: dispatch 2023-07-12T15:32:12.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:12.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:11 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:12.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-27"}]': finished 2023-07-12T15:32:12.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/512127389' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi069-133627-45","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:12.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:11 smithi069 ceph-mon[94241]: osdmap e219: 8 total, 8 up, 8 in 2023-07-12T15:32:12.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-27", "mode": "writeback"}]: dispatch 2023-07-12T15:32:12.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:12.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:11 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:13.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:12 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:13.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-27", "mode": "writeback"}]': finished 2023-07-12T15:32:13.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:12 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]': finished 2023-07-12T15:32:13.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:12 smithi145 ceph-mon[95801]: osdmap e220: 8 total, 8 up, 8 in 2023-07-12T15:32:13.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:13.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:12 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:13.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:13.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:13.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:13.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:12 smithi145 ceph-mon[95801]: pgmap v274: 228 pgs: 228 active+clean; 583 KiB data, 487 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2023-07-12T15:32:13.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:13.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-27", "mode": "writeback"}]': finished 2023-07-12T15:32:13.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]': finished 2023-07-12T15:32:13.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[102777]: osdmap e220: 8 total, 8 up, 8 in 2023-07-12T15:32:13.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:13.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:13.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:13.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:13.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:13.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[102777]: pgmap v274: 228 pgs: 228 active+clean; 583 KiB data, 487 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2023-07-12T15:32:13.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:13.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-27", "mode": "writeback"}]': finished 2023-07-12T15:32:13.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi069-133619-36"}]': finished 2023-07-12T15:32:13.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[94241]: osdmap e220: 8 total, 8 up, 8 in 2023-07-12T15:32:13.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2336863885' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:13.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]: dispatch 2023-07-12T15:32:13.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:13.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:13.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:13.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:12 smithi069 ceph-mon[94241]: pgmap v274: 228 pgs: 228 active+clean; 583 KiB data, 487 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2023-07-12T15:32:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]': finished 2023-07-12T15:32:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: osdmap e221: 8 total, 8 up, 8 in 2023-07-12T15:32:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:14.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:14.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:32:14.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "62.14", "id": [7, 5]}]: dispatch 2023-07-12T15:32:14.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "62.14", "id": [7, 5]}]: dispatch 2023-07-12T15:32:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]': finished 2023-07-12T15:32:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: osdmap e221: 8 total, 8 up, 8 in 2023-07-12T15:32:14.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:14.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:14.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:14.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:14.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:14.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:14.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:14.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:14.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:14.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:32:14.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "62.14", "id": [7, 5]}]: dispatch 2023-07-12T15:32:14.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "62.14", "id": [7, 5]}]: dispatch 2023-07-12T15:32:14.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi069-133619-36"}]': finished 2023-07-12T15:32:14.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:14.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: osdmap e221: 8 total, 8 up, 8 in 2023-07-12T15:32:14.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:14.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:14.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:14.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:14.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:14.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:14.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:14.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:14.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:14.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:32:14.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "62.14", "id": [7, 5]}]: dispatch 2023-07-12T15:32:14.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "62.14", "id": [7, 5]}]: dispatch 2023-07-12T15:32:15.077 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:32:14 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:32:15.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:15 smithi145 ceph-mon[95801]: pgmap v277: 228 pgs: 228 active+clean; 583 KiB data, 487 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:15 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "62.14", "id": [7, 5]}]': finished 2023-07-12T15:32:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:15 smithi145 ceph-mon[95801]: osdmap e222: 8 total, 8 up, 8 in 2023-07-12T15:32:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-133619-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:15.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-27"}]: dispatch 2023-07-12T15:32:15.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-133619-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:15.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:15 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:15.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:15 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:15.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-46"}]': finished 2023-07-12T15:32:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[102777]: pgmap v277: 228 pgs: 228 active+clean; 583 KiB data, 487 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "62.14", "id": [7, 5]}]': finished 2023-07-12T15:32:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[102777]: osdmap e222: 8 total, 8 up, 8 in 2023-07-12T15:32:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-133619-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-27"}]: dispatch 2023-07-12T15:32:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-133619-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-46"}]': finished 2023-07-12T15:32:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[94241]: pgmap v277: 228 pgs: 228 active+clean; 583 KiB data, 487 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "62.14", "id": [7, 5]}]': finished 2023-07-12T15:32:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[94241]: osdmap e222: 8 total, 8 up, 8 in 2023-07-12T15:32:15.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-133619-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:15.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-27"}]: dispatch 2023-07-12T15:32:15.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-133619-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:15.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:15.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:15.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-46"}]': finished 2023-07-12T15:32:16.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-27"}]': finished 2023-07-12T15:32:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:16 smithi145 ceph-mon[95801]: osdmap e223: 8 total, 8 up, 8 in 2023-07-12T15:32:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:16 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-133619-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-133619-37"}]': finished 2023-07-12T15:32:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:16 smithi145 ceph-mon[95801]: osdmap e224: 8 total, 8 up, 8 in 2023-07-12T15:32:16.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:32:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:32:16] "GET /metrics HTTP/1.1" 200 228244 "" "Prometheus/2.43.0" 2023-07-12T15:32:16.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-27"}]': finished 2023-07-12T15:32:16.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:16 smithi069 ceph-mon[102777]: osdmap e223: 8 total, 8 up, 8 in 2023-07-12T15:32:16.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:16 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-133619-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-133619-37"}]': finished 2023-07-12T15:32:16.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:16 smithi069 ceph-mon[102777]: osdmap e224: 8 total, 8 up, 8 in 2023-07-12T15:32:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-27"}]': finished 2023-07-12T15:32:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:16 smithi069 ceph-mon[94241]: osdmap e223: 8 total, 8 up, 8 in 2023-07-12T15:32:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:16 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi069-133619-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi069-133619-37"}]': finished 2023-07-12T15:32:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:16 smithi069 ceph-mon[94241]: osdmap e224: 8 total, 8 up, 8 in 2023-07-12T15:32:17.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:17 smithi145 ceph-mon[95801]: pgmap v280: 212 pgs: 16 unknown, 1 peering, 195 active+clean; 583 KiB data, 496 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:17 smithi145 ceph-mon[95801]: osdmap e225: 8 total, 8 up, 8 in 2023-07-12T15:32:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:17.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:17 smithi069 ceph-mon[102777]: pgmap v280: 212 pgs: 16 unknown, 1 peering, 195 active+clean; 583 KiB data, 496 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:17.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:17 smithi069 ceph-mon[102777]: osdmap e225: 8 total, 8 up, 8 in 2023-07-12T15:32:17.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:17.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:17.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:17 smithi069 ceph-mon[94241]: pgmap v280: 212 pgs: 16 unknown, 1 peering, 195 active+clean; 583 KiB data, 496 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:17.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:17 smithi069 ceph-mon[94241]: osdmap e225: 8 total, 8 up, 8 in 2023-07-12T15:32:17.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:17.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:19.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-29","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:19.106 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-46"}]': finished 2023-07-12T15:32:19.106 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:18 smithi145 ceph-mon[95801]: osdmap e226: 8 total, 8 up, 8 in 2023-07-12T15:32:19.107 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:19.107 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:19.108 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:18 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:19.109 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:18 smithi145 ceph-mon[95801]: pgmap v283: 228 pgs: 32 unknown, 1 peering, 195 active+clean; 583 KiB data, 496 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:19.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-29","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:19.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-46"}]': finished 2023-07-12T15:32:19.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[102777]: osdmap e226: 8 total, 8 up, 8 in 2023-07-12T15:32:19.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:19.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:19.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:19.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[102777]: pgmap v283: 228 pgs: 32 unknown, 1 peering, 195 active+clean; 583 KiB data, 496 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-29","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-46"}]': finished 2023-07-12T15:32:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[94241]: osdmap e226: 8 total, 8 up, 8 in 2023-07-12T15:32:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-46"}]: dispatch 2023-07-12T15:32:19.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:19.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:19.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:18 smithi069 ceph-mon[94241]: pgmap v283: 228 pgs: 32 unknown, 1 peering, 195 active+clean; 583 KiB data, 496 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:20.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-46"}]': finished 2023-07-12T15:32:20.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]': finished 2023-07-12T15:32:20.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[102777]: osdmap e227: 8 total, 8 up, 8 in 2023-07-12T15:32:20.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:20.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:20.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:20.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:20.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:20.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:20.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:20.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:20.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-46"}]': finished 2023-07-12T15:32:20.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]': finished 2023-07-12T15:32:20.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[94241]: osdmap e227: 8 total, 8 up, 8 in 2023-07-12T15:32:20.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:20.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:20.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:20.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:20.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:20.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:20.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:20.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:19 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:20.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2581656624' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-46"}]': finished 2023-07-12T15:32:20.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:19 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi069-133619-37"}]': finished 2023-07-12T15:32:20.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:19 smithi145 ceph-mon[95801]: osdmap e227: 8 total, 8 up, 8 in 2023-07-12T15:32:20.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/615008807' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:20.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:19 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]: dispatch 2023-07-12T15:32:20.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:20.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:19 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:20.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:20.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:19 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:20.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:20.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:19 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]': finished 2023-07-12T15:32:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: osdmap e228: 8 total, 8 up, 8 in 2023-07-12T15:32:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-29", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-133619-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-133619-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:32:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: pgmap v286: 228 pgs: 228 active+clean; 4.6 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s; 8 B/s, 4 objects/s recovering 2023-07-12T15:32:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:32:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:32:21.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:21.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]': finished 2023-07-12T15:32:21.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:21.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: osdmap e228: 8 total, 8 up, 8 in 2023-07-12T15:32:21.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:21.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-29", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-133619-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:21.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-133619-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:21.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:32:21.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: pgmap v286: 228 pgs: 228 active+clean; 4.6 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s; 8 B/s, 4 objects/s recovering 2023-07-12T15:32:21.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:32:21.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:32:21.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi069-133619-37"}]': finished 2023-07-12T15:32:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi069-133627-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: osdmap e228: 8 total, 8 up, 8 in 2023-07-12T15:32:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:21.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:21.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-29", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:21.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:21.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:21.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:21.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:21.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-133619-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:21.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-133619-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:21.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:32:21.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: pgmap v286: 228 pgs: 228 active+clean; 4.6 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s; 8 B/s, 4 objects/s recovering 2023-07-12T15:32:21.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:32:21.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:32:21.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:22.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-29", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:22.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-133619-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:22.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[102777]: osdmap e229: 8 total, 8 up, 8 in 2023-07-12T15:32:22.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-133619-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-29"}]: dispatch 2023-07-12T15:32:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-133619-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-47"}]': finished 2023-07-12T15:32:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-29"}]': finished 2023-07-12T15:32:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[102777]: osdmap e230: 8 total, 8 up, 8 in 2023-07-12T15:32:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-29", "mode": "writeback"}]: dispatch 2023-07-12T15:32:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-29", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-133619-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[94241]: osdmap e229: 8 total, 8 up, 8 in 2023-07-12T15:32:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-133619-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-29"}]: dispatch 2023-07-12T15:32:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-133619-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-47"}]': finished 2023-07-12T15:32:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-29"}]': finished 2023-07-12T15:32:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[94241]: osdmap e230: 8 total, 8 up, 8 in 2023-07-12T15:32:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-29", "mode": "writeback"}]: dispatch 2023-07-12T15:32:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-29", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:22 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi069-133619-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:22 smithi145 ceph-mon[95801]: osdmap e229: 8 total, 8 up, 8 in 2023-07-12T15:32:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-133619-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-29"}]: dispatch 2023-07-12T15:32:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:22 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-133619-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:22 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi069-133627-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi069-133627-47"}]': finished 2023-07-12T15:32:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-29"}]': finished 2023-07-12T15:32:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:22 smithi145 ceph-mon[95801]: osdmap e230: 8 total, 8 up, 8 in 2023-07-12T15:32:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-29", "mode": "writeback"}]: dispatch 2023-07-12T15:32:23.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:23 smithi069 ceph-mon[102777]: pgmap v289: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s; 8 B/s, 4 objects/s recovering 2023-07-12T15:32:23.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:23 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:23.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:23 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-133619-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-133619-38"}]': finished 2023-07-12T15:32:23.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-29", "mode": "writeback"}]': finished 2023-07-12T15:32:23.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:23 smithi069 ceph-mon[102777]: osdmap e231: 8 total, 8 up, 8 in 2023-07-12T15:32:23.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:23 smithi069 ceph-mon[94241]: pgmap v289: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s; 8 B/s, 4 objects/s recovering 2023-07-12T15:32:23.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:23 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:23.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:23 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-133619-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-133619-38"}]': finished 2023-07-12T15:32:23.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-29", "mode": "writeback"}]': finished 2023-07-12T15:32:23.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:23 smithi069 ceph-mon[94241]: osdmap e231: 8 total, 8 up, 8 in 2023-07-12T15:32:23.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:23 smithi145 ceph-mon[95801]: pgmap v289: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s; 8 B/s, 4 objects/s recovering 2023-07-12T15:32:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:23 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:23 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi069-133619-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi069-133619-38"}]': finished 2023-07-12T15:32:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-29", "mode": "writeback"}]': finished 2023-07-12T15:32:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:23 smithi145 ceph-mon[95801]: osdmap e231: 8 total, 8 up, 8 in 2023-07-12T15:32:25.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:32:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:32:25.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:24 smithi145 ceph-mon[95801]: osdmap e232: 8 total, 8 up, 8 in 2023-07-12T15:32:25.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:25.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:24 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:25.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:24 smithi145 ceph-mon[95801]: pgmap v292: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 526 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:25.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:24 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:25.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:24 smithi069 ceph-mon[102777]: osdmap e232: 8 total, 8 up, 8 in 2023-07-12T15:32:25.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:25.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:24 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:25.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:24 smithi069 ceph-mon[102777]: pgmap v292: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 526 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:25.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:24 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:25.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:24 smithi069 ceph-mon[94241]: osdmap e232: 8 total, 8 up, 8 in 2023-07-12T15:32:25.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:25.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:24 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:25.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:24 smithi069 ceph-mon[94241]: pgmap v292: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 526 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:25.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:24 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:26.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]': finished 2023-07-12T15:32:26.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:25 smithi145 ceph-mon[95801]: osdmap e233: 8 total, 8 up, 8 in 2023-07-12T15:32:26.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:26.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:26.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:26.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:26.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]': finished 2023-07-12T15:32:26.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:25 smithi069 ceph-mon[102777]: osdmap e233: 8 total, 8 up, 8 in 2023-07-12T15:32:26.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:26.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:26.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:26.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:26.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi069-133627-47"}]': finished 2023-07-12T15:32:26.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:25 smithi069 ceph-mon[94241]: osdmap e233: 8 total, 8 up, 8 in 2023-07-12T15:32:26.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2656115969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:26.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:26.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]: dispatch 2023-07-12T15:32:26.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:26.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:32:26] "GET /metrics HTTP/1.1" 200 228244 "" "Prometheus/2.43.0" 2023-07-12T15:32:27.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]': finished 2023-07-12T15:32:27.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]': finished 2023-07-12T15:32:27.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:26 smithi145 ceph-mon[95801]: osdmap e234: 8 total, 8 up, 8 in 2023-07-12T15:32:27.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:27.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:27.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:27.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:27.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:27.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:27.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-133627-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:27.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-133627-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:27.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:27.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:26 smithi145 ceph-mon[95801]: pgmap v295: 228 pgs: 228 active+clean; 8.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2023-07-12T15:32:27.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]': finished 2023-07-12T15:32:27.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]': finished 2023-07-12T15:32:27.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[102777]: osdmap e234: 8 total, 8 up, 8 in 2023-07-12T15:32:27.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:27.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:27.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:27.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:27.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:27.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:27.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-133627-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:27.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-133627-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:27.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:27.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[102777]: pgmap v295: 228 pgs: 228 active+clean; 8.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2023-07-12T15:32:27.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi069-133627-47"}]': finished 2023-07-12T15:32:27.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi069-133619-38"}]': finished 2023-07-12T15:32:27.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[94241]: osdmap e234: 8 total, 8 up, 8 in 2023-07-12T15:32:27.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1132606897' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:27.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]: dispatch 2023-07-12T15:32:27.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:27.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:27.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:27.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:27.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-133627-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:27.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-133627-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:27.241 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:27.242 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:26 smithi069 ceph-mon[94241]: pgmap v295: 228 pgs: 228 active+clean; 8.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2023-07-12T15:32:28.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:28 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]': finished 2023-07-12T15:32:28.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:28 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-133627-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:28.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:28.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:28 smithi145 ceph-mon[95801]: osdmap e235: 8 total, 8 up, 8 in 2023-07-12T15:32:28.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-29"}]: dispatch 2023-07-12T15:32:28.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-133627-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:28.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:28 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-133627-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:28.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:28.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:28.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-133619-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:28.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]': finished 2023-07-12T15:32:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-133627-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[102777]: osdmap e235: 8 total, 8 up, 8 in 2023-07-12T15:32:28.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-29"}]: dispatch 2023-07-12T15:32:28.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-133627-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:28.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-133627-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:28.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:28.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:28.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-133619-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:28.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi069-133619-38"}]': finished 2023-07-12T15:32:28.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi069-133627-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:28.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:28.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[94241]: osdmap e235: 8 total, 8 up, 8 in 2023-07-12T15:32:28.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-29"}]: dispatch 2023-07-12T15:32:28.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-133627-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:28.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-133627-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:28.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:28.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:28.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-133619-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:29 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-29"}]': finished 2023-07-12T15:32:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:29 smithi145 ceph-mon[95801]: pgmap v298: 228 pgs: 228 active+clean; 8.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2023-07-12T15:32:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-133619-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:29 smithi145 ceph-mon[95801]: osdmap e236: 8 total, 8 up, 8 in 2023-07-12T15:32:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-133619-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:29.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:29 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:29.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:32:29.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:29 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-133627-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-133627-48"}]': finished 2023-07-12T15:32:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-29"}]': finished 2023-07-12T15:32:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[94241]: pgmap v298: 228 pgs: 228 active+clean; 8.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2023-07-12T15:32:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-133619-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[94241]: osdmap e236: 8 total, 8 up, 8 in 2023-07-12T15:32:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-133619-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:32:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-133627-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-133627-48"}]': finished 2023-07-12T15:32:29.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:29.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-29"}]': finished 2023-07-12T15:32:29.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[102777]: pgmap v298: 228 pgs: 228 active+clean; 8.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2023-07-12T15:32:29.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi069-133619-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:29.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[102777]: osdmap e236: 8 total, 8 up, 8 in 2023-07-12T15:32:29.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-133619-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:29.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:29.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:32:29.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:29 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi069-133627-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi069-133627-48"}]': finished 2023-07-12T15:32:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:30 smithi145 ceph-mon[95801]: osdmap e237: 8 total, 8 up, 8 in 2023-07-12T15:32:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-133619-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-133619-39"}]': finished 2023-07-12T15:32:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:30 smithi145 ceph-mon[95801]: osdmap e238: 8 total, 8 up, 8 in 2023-07-12T15:32:30.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:30.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:30 smithi069 ceph-mon[102777]: osdmap e237: 8 total, 8 up, 8 in 2023-07-12T15:32:30.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-133619-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-133619-39"}]': finished 2023-07-12T15:32:30.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:30 smithi069 ceph-mon[102777]: osdmap e238: 8 total, 8 up, 8 in 2023-07-12T15:32:30.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:30.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:30 smithi069 ceph-mon[94241]: osdmap e237: 8 total, 8 up, 8 in 2023-07-12T15:32:30.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi069-133619-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi069-133619-39"}]': finished 2023-07-12T15:32:30.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:30 smithi069 ceph-mon[94241]: osdmap e238: 8 total, 8 up, 8 in 2023-07-12T15:32:30.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:31.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:31 smithi145 ceph-mon[95801]: pgmap v301: 244 pgs: 4 creating+peering, 44 unknown, 196 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:31.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-31","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:31.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:31 smithi145 ceph-mon[95801]: osdmap e239: 8 total, 8 up, 8 in 2023-07-12T15:32:31.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:31.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-31", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:31.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:31 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:31.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:31 smithi069 ceph-mon[102777]: pgmap v301: 244 pgs: 4 creating+peering, 44 unknown, 196 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-31","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:31 smithi069 ceph-mon[102777]: osdmap e239: 8 total, 8 up, 8 in 2023-07-12T15:32:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-31", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:31 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:31.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:31 smithi069 ceph-mon[94241]: pgmap v301: 244 pgs: 4 creating+peering, 44 unknown, 196 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:31.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-31","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:31.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:31 smithi069 ceph-mon[94241]: osdmap e239: 8 total, 8 up, 8 in 2023-07-12T15:32:31.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:31.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-31", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:31.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:31 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:33.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-31", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:33.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:32 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]': finished 2023-07-12T15:32:33.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:32 smithi145 ceph-mon[95801]: osdmap e240: 8 total, 8 up, 8 in 2023-07-12T15:32:33.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-31"}]: dispatch 2023-07-12T15:32:33.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:33.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:33.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:32 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:33.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:32 smithi145 ceph-mon[95801]: pgmap v304: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:33.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-31", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:33.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]': finished 2023-07-12T15:32:33.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[102777]: osdmap e240: 8 total, 8 up, 8 in 2023-07-12T15:32:33.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-31"}]: dispatch 2023-07-12T15:32:33.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:33.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:33.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:33.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[102777]: pgmap v304: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:33.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-31", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:33.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi069-133627-48"}]': finished 2023-07-12T15:32:33.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[94241]: osdmap e240: 8 total, 8 up, 8 in 2023-07-12T15:32:33.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-31"}]: dispatch 2023-07-12T15:32:33.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1947501830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:33.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:33.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]: dispatch 2023-07-12T15:32:33.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:32 smithi069 ceph-mon[94241]: pgmap v304: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:34.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-31"}]': finished 2023-07-12T15:32:34.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-133619-39"}]': finished 2023-07-12T15:32:34.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]': finished 2023-07-12T15:32:34.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: osdmap e241: 8 total, 8 up, 8 in 2023-07-12T15:32:34.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:34.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-31", "mode": "writeback"}]: dispatch 2023-07-12T15:32:34.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:34.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:34.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:34.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:34.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-133627-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:34.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-133627-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:34.104 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:34.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-133619-39"}]': finished 2023-07-12T15:32:34.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-31", "mode": "writeback"}]': finished 2023-07-12T15:32:34.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-133627-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:34.105 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:33 smithi145 ceph-mon[95801]: osdmap e242: 8 total, 8 up, 8 in 2023-07-12T15:32:34.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-31"}]': finished 2023-07-12T15:32:34.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-133619-39"}]': finished 2023-07-12T15:32:34.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]': finished 2023-07-12T15:32:34.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: osdmap e241: 8 total, 8 up, 8 in 2023-07-12T15:32:34.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:34.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-31", "mode": "writeback"}]: dispatch 2023-07-12T15:32:34.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:34.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:34.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:34.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:34.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-133627-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:34.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-133627-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:34.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:34.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-133619-39"}]': finished 2023-07-12T15:32:34.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-31", "mode": "writeback"}]': finished 2023-07-12T15:32:34.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-133627-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:34.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[94241]: osdmap e242: 8 total, 8 up, 8 in 2023-07-12T15:32:34.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-31"}]': finished 2023-07-12T15:32:34.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi069-133619-39"}]': finished 2023-07-12T15:32:34.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi069-133627-48"}]': finished 2023-07-12T15:32:34.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: osdmap e241: 8 total, 8 up, 8 in 2023-07-12T15:32:34.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-133619-39"}]: dispatch 2023-07-12T15:32:34.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-31", "mode": "writeback"}]: dispatch 2023-07-12T15:32:34.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:34.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:34.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:34.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:34.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-133627-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:34.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-133627-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:34.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:34.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1957060471' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi069-133619-39"}]': finished 2023-07-12T15:32:34.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-31", "mode": "writeback"}]': finished 2023-07-12T15:32:34.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi069-133627-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:34.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:33 smithi069 ceph-mon[102777]: osdmap e242: 8 total, 8 up, 8 in 2023-07-12T15:32:35.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:32:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:32:35.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-133627-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:35.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:34 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-133627-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:35.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:35.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:35.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi069-133619-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:35.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:35.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:34 smithi145 ceph-mon[95801]: pgmap v307: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-133627-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-133627-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:35.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:35.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi069-133619-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:35.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:35.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[102777]: pgmap v307: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:35.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-133627-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:35.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-133627-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:35.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:35.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:35.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi069-133619-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:35.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:35.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:34 smithi069 ceph-mon[94241]: pgmap v307: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:35 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi069-133619-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:35 smithi145 ceph-mon[95801]: osdmap e243: 8 total, 8 up, 8 in 2023-07-12T15:32:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-31"}]: dispatch 2023-07-12T15:32:36.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi069-133619-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:36.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:35 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:36.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi069-133619-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:36.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:36.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:35 smithi069 ceph-mon[102777]: osdmap e243: 8 total, 8 up, 8 in 2023-07-12T15:32:36.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-31"}]: dispatch 2023-07-12T15:32:36.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi069-133619-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:36.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:35 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:36.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi069-133619-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:36.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:36.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:35 smithi069 ceph-mon[94241]: osdmap e243: 8 total, 8 up, 8 in 2023-07-12T15:32:36.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-31"}]: dispatch 2023-07-12T15:32:36.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi069-133619-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:36.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:32:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:32:36] "GET /metrics HTTP/1.1" 200 228338 "" "Prometheus/2.43.0" 2023-07-12T15:32:37.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:36 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:36 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-133627-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-133627-49"}]': finished 2023-07-12T15:32:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-31"}]': finished 2023-07-12T15:32:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:36 smithi145 ceph-mon[95801]: pgmap v310: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:32:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:36 smithi145 ceph-mon[95801]: osdmap e244: 8 total, 8 up, 8 in 2023-07-12T15:32:37.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:36 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:37.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:36 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-133627-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-133627-49"}]': finished 2023-07-12T15:32:37.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-31"}]': finished 2023-07-12T15:32:37.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:36 smithi069 ceph-mon[94241]: pgmap v310: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:32:37.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:36 smithi069 ceph-mon[94241]: osdmap e244: 8 total, 8 up, 8 in 2023-07-12T15:32:37.244 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:36 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:37.245 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:36 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi069-133627-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi069-133627-49"}]': finished 2023-07-12T15:32:37.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-31"}]': finished 2023-07-12T15:32:37.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:36 smithi069 ceph-mon[102777]: pgmap v310: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:32:37.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:36 smithi069 ceph-mon[102777]: osdmap e244: 8 total, 8 up, 8 in 2023-07-12T15:32:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi069-133619-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi069-133619-40"}]': finished 2023-07-12T15:32:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:37 smithi145 ceph-mon[95801]: osdmap e245: 8 total, 8 up, 8 in 2023-07-12T15:32:38.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi069-133619-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi069-133619-40"}]': finished 2023-07-12T15:32:38.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:37 smithi069 ceph-mon[102777]: osdmap e245: 8 total, 8 up, 8 in 2023-07-12T15:32:38.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi069-133619-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi069-133619-40"}]': finished 2023-07-12T15:32:38.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:37 smithi069 ceph-mon[94241]: osdmap e245: 8 total, 8 up, 8 in 2023-07-12T15:32:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:38 smithi145 ceph-mon[95801]: osdmap e246: 8 total, 8 up, 8 in 2023-07-12T15:32:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:39.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:38 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:39.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:38 smithi145 ceph-mon[95801]: pgmap v313: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:32:39.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-33","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:39.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:38 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]': finished 2023-07-12T15:32:39.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:38 smithi145 ceph-mon[95801]: osdmap e247: 8 total, 8 up, 8 in 2023-07-12T15:32:39.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[102777]: osdmap e246: 8 total, 8 up, 8 in 2023-07-12T15:32:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[102777]: pgmap v313: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:32:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-33","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]': finished 2023-07-12T15:32:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[102777]: osdmap e247: 8 total, 8 up, 8 in 2023-07-12T15:32:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:39.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[94241]: osdmap e246: 8 total, 8 up, 8 in 2023-07-12T15:32:39.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:39.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:39.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:39.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[94241]: pgmap v313: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:32:39.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-33","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:39.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi069-133627-49"}]': finished 2023-07-12T15:32:39.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[94241]: osdmap e247: 8 total, 8 up, 8 in 2023-07-12T15:32:39.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/670805027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:40.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:40.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:40.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-33", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:40.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]': finished 2023-07-12T15:32:40.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-133619-40"}]': finished 2023-07-12T15:32:40.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-33", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:40.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: osdmap e248: 8 total, 8 up, 8 in 2023-07-12T15:32:40.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-33"}]: dispatch 2023-07-12T15:32:40.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:40.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:40.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:40.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:40.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:40.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-133627-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:40.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:39 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-133627-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:40.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:40.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:40.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-33", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:40.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]': finished 2023-07-12T15:32:40.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-133619-40"}]': finished 2023-07-12T15:32:40.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-33", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:40.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: osdmap e248: 8 total, 8 up, 8 in 2023-07-12T15:32:40.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-33"}]: dispatch 2023-07-12T15:32:40.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:40.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:40.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:40.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:40.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:40.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-133627-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:40.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-133627-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:40.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]: dispatch 2023-07-12T15:32:40.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:40.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-33", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:40.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi069-133627-49"}]': finished 2023-07-12T15:32:40.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi069-133619-40"}]': finished 2023-07-12T15:32:40.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-33", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:40.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: osdmap e248: 8 total, 8 up, 8 in 2023-07-12T15:32:40.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-33"}]: dispatch 2023-07-12T15:32:40.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-133619-40"}]: dispatch 2023-07-12T15:32:40.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:40.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:40.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:40.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:40.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-133627-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:40.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:39 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-133627-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:41.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: pgmap v316: 228 pgs: 228 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:32:41.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-33"}]': finished 2023-07-12T15:32:41.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-133619-40"}]': finished 2023-07-12T15:32:41.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-133627-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:41.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: osdmap e249: 8 total, 8 up, 8 in 2023-07-12T15:32:41.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-33", "mode": "writeback"}]: dispatch 2023-07-12T15:32:41.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-133627-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:41.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-133627-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:41.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:41.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:41.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:41.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:41.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-133619-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:41.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-133619-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:41.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: pgmap v316: 228 pgs: 228 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:32:41.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-33"}]': finished 2023-07-12T15:32:41.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-133619-40"}]': finished 2023-07-12T15:32:41.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-133627-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:41.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: osdmap e249: 8 total, 8 up, 8 in 2023-07-12T15:32:41.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-33", "mode": "writeback"}]: dispatch 2023-07-12T15:32:41.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-133627-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:41.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-133627-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:41.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:41.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:41.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:41.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:41.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-133619-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:41.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:40 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-133619-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: pgmap v316: 228 pgs: 228 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:32:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-33"}]': finished 2023-07-12T15:32:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3760129548' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi069-133619-40"}]': finished 2023-07-12T15:32:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi069-133627-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: osdmap e249: 8 total, 8 up, 8 in 2023-07-12T15:32:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-33", "mode": "writeback"}]: dispatch 2023-07-12T15:32:41.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-133627-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:41.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-133627-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:41.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:41.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:41.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:41.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:41.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-133619-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:41.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:40 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-133619-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:43.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:42 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-33", "mode": "writeback"}]': finished 2023-07-12T15:32:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:42 smithi145 ceph-mon[95801]: pgmap v319: 228 pgs: 228 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:32:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:42 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-133619-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:42 smithi145 ceph-mon[95801]: osdmap e250: 8 total, 8 up, 8 in 2023-07-12T15:32:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-133619-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:43.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:42 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-133619-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:43.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-33", "mode": "writeback"}]': finished 2023-07-12T15:32:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[102777]: pgmap v319: 228 pgs: 228 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:32:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-133619-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:43.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[102777]: osdmap e250: 8 total, 8 up, 8 in 2023-07-12T15:32:43.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-133619-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:43.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-133619-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:43.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:43.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:43.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-33", "mode": "writeback"}]': finished 2023-07-12T15:32:43.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[94241]: pgmap v319: 228 pgs: 228 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:32:43.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi069-133619-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:43.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[94241]: osdmap e250: 8 total, 8 up, 8 in 2023-07-12T15:32:43.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-133619-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:43.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-133619-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:43.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:43 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-133627-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-133627-50"}]': finished 2023-07-12T15:32:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:43 smithi145 ceph-mon[95801]: osdmap e251: 8 total, 8 up, 8 in 2023-07-12T15:32:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-33"}]: dispatch 2023-07-12T15:32:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:43 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:43 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:32:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:43 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-133627-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-133627-50"}]': finished 2023-07-12T15:32:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:43 smithi069 ceph-mon[102777]: osdmap e251: 8 total, 8 up, 8 in 2023-07-12T15:32:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-33"}]: dispatch 2023-07-12T15:32:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:43 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:44.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:43 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:32:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:43 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi069-133627-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi069-133627-50"}]': finished 2023-07-12T15:32:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:43 smithi069 ceph-mon[94241]: osdmap e251: 8 total, 8 up, 8 in 2023-07-12T15:32:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-33"}]: dispatch 2023-07-12T15:32:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:43 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:43 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:32:45.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:32:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:32:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:44 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:44 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-133619-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-133619-41"}]': finished 2023-07-12T15:32:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-33"}]': finished 2023-07-12T15:32:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:44 smithi145 ceph-mon[95801]: osdmap e252: 8 total, 8 up, 8 in 2023-07-12T15:32:45.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:44 smithi145 ceph-mon[95801]: pgmap v322: 244 pgs: 16 unknown, 228 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:45.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:44 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:44 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-133619-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-133619-41"}]': finished 2023-07-12T15:32:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-33"}]': finished 2023-07-12T15:32:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:44 smithi069 ceph-mon[102777]: osdmap e252: 8 total, 8 up, 8 in 2023-07-12T15:32:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:44 smithi069 ceph-mon[102777]: pgmap v322: 244 pgs: 16 unknown, 228 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:44 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:44 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi069-133619-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi069-133619-41"}]': finished 2023-07-12T15:32:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-33"}]': finished 2023-07-12T15:32:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:44 smithi069 ceph-mon[94241]: osdmap e252: 8 total, 8 up, 8 in 2023-07-12T15:32:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:44 smithi069 ceph-mon[94241]: pgmap v322: 244 pgs: 16 unknown, 228 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:45 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:45 smithi145 ceph-mon[95801]: osdmap e253: 8 total, 8 up, 8 in 2023-07-12T15:32:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:45 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:46.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:45 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:46.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:45 smithi069 ceph-mon[102777]: osdmap e253: 8 total, 8 up, 8 in 2023-07-12T15:32:46.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:46.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:45 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:46.149 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:32:46] "GET /metrics HTTP/1.1" 200 230722 "" "Prometheus/2.43.0" 2023-07-12T15:32:46.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:45 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:46.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:45 smithi069 ceph-mon[94241]: osdmap e253: 8 total, 8 up, 8 in 2023-07-12T15:32:46.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:46.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:45 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:47.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]': finished 2023-07-12T15:32:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[94241]: osdmap e254: 8 total, 8 up, 8 in 2023-07-12T15:32:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:47.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:47.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:47.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:47.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[94241]: pgmap v325: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:32:47.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]': finished 2023-07-12T15:32:47.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[102777]: osdmap e254: 8 total, 8 up, 8 in 2023-07-12T15:32:47.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:47.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:47.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:47.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:47.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:47.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:46 smithi069 ceph-mon[102777]: pgmap v325: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:32:47.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi069-133627-50"}]': finished 2023-07-12T15:32:47.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:46 smithi145 ceph-mon[95801]: osdmap e254: 8 total, 8 up, 8 in 2023-07-12T15:32:47.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1749209096' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:47.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:47.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]: dispatch 2023-07-12T15:32:47.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:47.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:47.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:46 smithi145 ceph-mon[95801]: pgmap v325: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:32:48.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]': finished 2023-07-12T15:32:48.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]': finished 2023-07-12T15:32:48.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-35","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:48.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[102777]: osdmap e255: 8 total, 8 up, 8 in 2023-07-12T15:32:48.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-35", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:48.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:48.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:48.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]': finished 2023-07-12T15:32:48.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]': finished 2023-07-12T15:32:48.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-35","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:48.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[94241]: osdmap e255: 8 total, 8 up, 8 in 2023-07-12T15:32:48.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-35", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:48.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:48.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:47 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:48.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi069-133627-50"}]': finished 2023-07-12T15:32:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi069-133619-41"}]': finished 2023-07-12T15:32:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-35","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:47 smithi145 ceph-mon[95801]: osdmap e255: 8 total, 8 up, 8 in 2023-07-12T15:32:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-35", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:32:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3350723845' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:48.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:47 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]: dispatch 2023-07-12T15:32:49.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-35", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:49.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]': finished 2023-07-12T15:32:49.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[102777]: osdmap e256: 8 total, 8 up, 8 in 2023-07-12T15:32:49.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-35"}]: dispatch 2023-07-12T15:32:49.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4165981073' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:49.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-133619-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-133619-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[102777]: pgmap v328: 260 pgs: 64 unknown, 196 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:32:49.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-35", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:49.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]': finished 2023-07-12T15:32:49.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[94241]: osdmap e256: 8 total, 8 up, 8 in 2023-07-12T15:32:49.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-35"}]: dispatch 2023-07-12T15:32:49.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4165981073' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:49.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:49.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:49.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:49.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:49.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-133619-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:49.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-133619-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:49.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:48 smithi069 ceph-mon[94241]: pgmap v328: 260 pgs: 64 unknown, 196 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:32:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-35", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:32:49.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi069-133619-41"}]': finished 2023-07-12T15:32:49.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:48 smithi145 ceph-mon[95801]: osdmap e256: 8 total, 8 up, 8 in 2023-07-12T15:32:49.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-35"}]: dispatch 2023-07-12T15:32:49.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4165981073' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:49.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:49.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:49.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:49.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:49.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-133619-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:49.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-133619-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:49.355 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:48 smithi145 ceph-mon[95801]: pgmap v328: 260 pgs: 64 unknown, 196 active+clean; 4.6 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:32:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-35"}]': finished 2023-07-12T15:32:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4165981073' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-51","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-133619-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[102777]: osdmap e257: 8 total, 8 up, 8 in 2023-07-12T15:32:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-133619-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-35", "mode": "writeback"}]: dispatch 2023-07-12T15:32:50.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-133619-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:50.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:50.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-35"}]': finished 2023-07-12T15:32:50.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4165981073' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-51","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:50.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-133619-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:50.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[94241]: osdmap e257: 8 total, 8 up, 8 in 2023-07-12T15:32:50.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-133619-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:50.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-35", "mode": "writeback"}]: dispatch 2023-07-12T15:32:50.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-133619-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:50.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:50 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:50.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-35"}]': finished 2023-07-12T15:32:50.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4165981073' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi069-133627-51","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:32:50.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi069-133619-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:50.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:50 smithi145 ceph-mon[95801]: osdmap e257: 8 total, 8 up, 8 in 2023-07-12T15:32:50.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-133619-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:50.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-35", "mode": "writeback"}]: dispatch 2023-07-12T15:32:50.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-133619-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:50.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:50 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:32:51.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-35", "mode": "writeback"}]': finished 2023-07-12T15:32:51.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[94241]: pgmap v331: 228 pgs: 228 active+clean; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:51.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[94241]: osdmap e258: 8 total, 8 up, 8 in 2023-07-12T15:32:51.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:51.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:51.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:51.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-133619-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-133619-42"}]': finished 2023-07-12T15:32:51.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:51.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[94241]: osdmap e259: 8 total, 8 up, 8 in 2023-07-12T15:32:51.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi069-133627-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:51.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-35", "mode": "writeback"}]': finished 2023-07-12T15:32:51.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[102777]: pgmap v331: 228 pgs: 228 active+clean; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:51.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[102777]: osdmap e258: 8 total, 8 up, 8 in 2023-07-12T15:32:51.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:51.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:51.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:51.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-133619-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-133619-42"}]': finished 2023-07-12T15:32:51.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:51.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[102777]: osdmap e259: 8 total, 8 up, 8 in 2023-07-12T15:32:51.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi069-133627-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:51.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-35", "mode": "writeback"}]': finished 2023-07-12T15:32:51.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:51 smithi145 ceph-mon[95801]: pgmap v331: 228 pgs: 228 active+clean; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:51.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:51 smithi145 ceph-mon[95801]: osdmap e258: 8 total, 8 up, 8 in 2023-07-12T15:32:51.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:51.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:51.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:51.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi069-133619-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi069-133619-42"}]': finished 2023-07-12T15:32:51.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:51.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:51 smithi145 ceph-mon[95801]: osdmap e259: 8 total, 8 up, 8 in 2023-07-12T15:32:51.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi069-133627-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:52 smithi069 ceph-mon[102777]: pgmap v334: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:52 smithi069 ceph-mon[102777]: osdmap e260: 8 total, 8 up, 8 in 2023-07-12T15:32:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:52 smithi069 ceph-mon[94241]: pgmap v334: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:52 smithi069 ceph-mon[94241]: osdmap e260: 8 total, 8 up, 8 in 2023-07-12T15:32:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:53.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:52 smithi145 ceph-mon[95801]: pgmap v334: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:32:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:52 smithi145 ceph-mon[95801]: osdmap e260: 8 total, 8 up, 8 in 2023-07-12T15:32:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:54.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi069-133627-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]': finished 2023-07-12T15:32:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:53 smithi069 ceph-mon[102777]: osdmap e261: 8 total, 8 up, 8 in 2023-07-12T15:32:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-35"}]: dispatch 2023-07-12T15:32:54.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi069-133627-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]': finished 2023-07-12T15:32:54.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:54.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:53 smithi069 ceph-mon[94241]: osdmap e261: 8 total, 8 up, 8 in 2023-07-12T15:32:54.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:54.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:54.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-35"}]: dispatch 2023-07-12T15:32:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi069-133627-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]': finished 2023-07-12T15:32:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:53 smithi145 ceph-mon[95801]: osdmap e261: 8 total, 8 up, 8 in 2023-07-12T15:32:54.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:54.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:54.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-35"}]: dispatch 2023-07-12T15:32:54.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7042 ms) 2023-07-12T15:32:54.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-07-12T15:32:54.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6920 ms) 2023-07-12T15:32:54.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-07-12T15:32:54.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6993 ms) 2023-07-12T15:32:54.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-07-12T15:32:54.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7042 ms) 2023-07-12T15:32:54.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-07-12T15:32:54.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7009 ms) 2023-07-12T15:32:54.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-07-12T15:32:54.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7041 ms) 2023-07-12T15:32:54.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-07-12T15:32:54.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7023 ms) 2023-07-12T15:32:54.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-07-12T15:32:54.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7035 ms) 2023-07-12T15:32:54.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-07-12T15:32:54.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7109 ms) 2023-07-12T15:32:54.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-07-12T15:32:54.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6807 ms) 2023-07-12T15:32:54.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-07-12T15:32:54.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7026 ms) 2023-07-12T15:32:54.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-07-12T15:32:54.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6936 ms) 2023-07-12T15:32:54.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-07-12T15:32:54.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6985 ms) 2023-07-12T15:32:54.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-07-12T15:32:54.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7037 ms) 2023-07-12T15:32:54.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-07-12T15:32:54.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7050 ms) 2023-07-12T15:32:54.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-07-12T15:32:54.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7023 ms) 2023-07-12T15:32:54.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112078 ms total) 2023-07-12T15:32:54.908 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: 2023-07-12T15:32:54.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] Global test environment tear-down 2023-07-12T15:32:54.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (194256 ms total) 2023-07-12T15:32:54.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ PASSED ] 42 tests. 2023-07-12T15:32:54.909 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:32:54 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:32:55.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]': finished 2023-07-12T15:32:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-35"}]': finished 2023-07-12T15:32:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[102777]: osdmap e262: 8 total, 8 up, 8 in 2023-07-12T15:32:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[102777]: pgmap v337: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]': finished 2023-07-12T15:32:55.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[102777]: osdmap e263: 8 total, 8 up, 8 in 2023-07-12T15:32:55.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:55.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]': finished 2023-07-12T15:32:55.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-35"}]': finished 2023-07-12T15:32:55.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[94241]: osdmap e262: 8 total, 8 up, 8 in 2023-07-12T15:32:55.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:55.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:55.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[94241]: pgmap v337: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:55.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]': finished 2023-07-12T15:32:55.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[94241]: osdmap e263: 8 total, 8 up, 8 in 2023-07-12T15:32:55.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:55.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi069-133619-42"}]': finished 2023-07-12T15:32:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-35"}]': finished 2023-07-12T15:32:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:54 smithi145 ceph-mon[95801]: osdmap e262: 8 total, 8 up, 8 in 2023-07-12T15:32:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/116936424' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]: dispatch 2023-07-12T15:32:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:54 smithi145 ceph-mon[95801]: pgmap v337: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:32:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi069-133619-42"}]': finished 2023-07-12T15:32:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:54 smithi145 ceph-mon[95801]: osdmap e263: 8 total, 8 up, 8 in 2023-07-12T15:32:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:56.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:32:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:32:56] "GET /metrics HTTP/1.1" 200 230722 "" "Prometheus/2.43.0" 2023-07-12T15:32:56.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:56.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:56.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]': finished 2023-07-12T15:32:56.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:56.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[94241]: osdmap e264: 8 total, 8 up, 8 in 2023-07-12T15:32:56.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:56.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-35"}]: dispatch 2023-07-12T15:32:56.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:56.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:56.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]': finished 2023-07-12T15:32:56.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:56.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[102777]: osdmap e264: 8 total, 8 up, 8 in 2023-07-12T15:32:56.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:56.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-35"}]: dispatch 2023-07-12T15:32:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:55 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:32:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:32:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi069-133627-52"}]': finished 2023-07-12T15:32:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:32:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:55 smithi145 ceph-mon[95801]: osdmap e264: 8 total, 8 up, 8 in 2023-07-12T15:32:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-133627-52"}]: dispatch 2023-07-12T15:32:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-35"}]: dispatch 2023-07-12T15:32:57.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:56 smithi069 ceph-mon[102777]: pgmap v340: 228 pgs: 228 active+clean; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-07-12T15:32:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:56 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-133627-52"}]': finished 2023-07-12T15:32:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-35"}]': finished 2023-07-12T15:32:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:56 smithi069 ceph-mon[102777]: osdmap e265: 8 total, 8 up, 8 in 2023-07-12T15:32:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:56 smithi069 ceph-mon[94241]: pgmap v340: 228 pgs: 228 active+clean; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-07-12T15:32:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:56 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-133627-52"}]': finished 2023-07-12T15:32:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-35"}]': finished 2023-07-12T15:32:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:56 smithi069 ceph-mon[94241]: osdmap e265: 8 total, 8 up, 8 in 2023-07-12T15:32:57.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:56 smithi145 ceph-mon[95801]: pgmap v340: 228 pgs: 228 active+clean; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-07-12T15:32:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:56 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:32:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1275076393' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi069-133627-52"}]': finished 2023-07-12T15:32:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-35"}]': finished 2023-07-12T15:32:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:56 smithi145 ceph-mon[95801]: osdmap e265: 8 total, 8 up, 8 in 2023-07-12T15:32:58.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:32:58.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:32:58.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:58.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:32:58.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:32:58.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:58.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:32:58.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:32:58.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:32:59.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:59.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[102777]: osdmap e266: 8 total, 8 up, 8 in 2023-07-12T15:32:59.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi069-133627-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:32:59.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[102777]: pgmap v343: 196 pgs: 196 active+clean; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:32:59.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:59.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:32:59.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[102777]: osdmap e267: 8 total, 8 up, 8 in 2023-07-12T15:32:59.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:59.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:59.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[94241]: osdmap e266: 8 total, 8 up, 8 in 2023-07-12T15:32:59.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi069-133627-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:32:59.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[94241]: pgmap v343: 196 pgs: 196 active+clean; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:32:59.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:59.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:32:59.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[94241]: osdmap e267: 8 total, 8 up, 8 in 2023-07-12T15:32:59.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:32:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:32:59.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:32:59.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:59 smithi145 ceph-mon[95801]: osdmap e266: 8 total, 8 up, 8 in 2023-07-12T15:32:59.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi069-133627-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:32:59.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:59 smithi145 ceph-mon[95801]: pgmap v343: 196 pgs: 196 active+clean; 4.6 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:32:59.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:59 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:32:59.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:59 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:32:59.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:59 smithi145 ceph-mon[95801]: osdmap e267: 8 total, 8 up, 8 in 2023-07-12T15:32:59.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:32:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:00.931 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi069-133627-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi069-133627-53"}]': finished 2023-07-12T15:33:00.932 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-37","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:01.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi069-133627-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi069-133627-53"}]': finished 2023-07-12T15:33:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-37","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:00 smithi069 ceph-mon[102777]: osdmap e268: 8 total, 8 up, 8 in 2023-07-12T15:33:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-37", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:00 smithi069 ceph-mon[102777]: pgmap v346: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 4.6 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2023-07-12T15:33:01.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi069-133627-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi069-133627-53"}]': finished 2023-07-12T15:33:01.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-37","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:01.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:00 smithi069 ceph-mon[94241]: osdmap e268: 8 total, 8 up, 8 in 2023-07-12T15:33:01.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-37", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:01.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:00 smithi069 ceph-mon[94241]: pgmap v346: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 4.6 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2023-07-12T15:33:01.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:00 smithi145 ceph-mon[95801]: osdmap e268: 8 total, 8 up, 8 in 2023-07-12T15:33:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-37", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:00 smithi145 ceph-mon[95801]: pgmap v346: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 4.6 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2023-07-12T15:33:02.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-37", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:02.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[94241]: osdmap e269: 8 total, 8 up, 8 in 2023-07-12T15:33:02.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-37"}]: dispatch 2023-07-12T15:33:02.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-37"}]': finished 2023-07-12T15:33:02.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[94241]: osdmap e270: 8 total, 8 up, 8 in 2023-07-12T15:33:02.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:33:02.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-37", "mode": "writeback"}]: dispatch 2023-07-12T15:33:02.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-37", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:02.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[102777]: osdmap e269: 8 total, 8 up, 8 in 2023-07-12T15:33:02.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-37"}]: dispatch 2023-07-12T15:33:02.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-37"}]': finished 2023-07-12T15:33:02.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[102777]: osdmap e270: 8 total, 8 up, 8 in 2023-07-12T15:33:02.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:33:02.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-37", "mode": "writeback"}]: dispatch 2023-07-12T15:33:02.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-37", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:02.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:01 smithi145 ceph-mon[95801]: osdmap e269: 8 total, 8 up, 8 in 2023-07-12T15:33:02.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-37"}]: dispatch 2023-07-12T15:33:02.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-37"}]': finished 2023-07-12T15:33:02.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:01 smithi145 ceph-mon[95801]: osdmap e270: 8 total, 8 up, 8 in 2023-07-12T15:33:02.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:33:02.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-37", "mode": "writeback"}]: dispatch 2023-07-12T15:33:03.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[94241]: pgmap v349: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:33:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53"}]': finished 2023-07-12T15:33:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-37", "mode": "writeback"}]': finished 2023-07-12T15:33:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[94241]: osdmap e271: 8 total, 8 up, 8 in 2023-07-12T15:33:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:33:03.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[102777]: pgmap v349: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:33:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53"}]': finished 2023-07-12T15:33:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-37", "mode": "writeback"}]': finished 2023-07-12T15:33:03.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[102777]: osdmap e271: 8 total, 8 up, 8 in 2023-07-12T15:33:03.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:33:03.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:03.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:02 smithi145 ceph-mon[95801]: pgmap v349: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:03.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:02 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:33:03.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi069-133627-53"}]': finished 2023-07-12T15:33:03.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-37", "mode": "writeback"}]': finished 2023-07-12T15:33:03.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:02 smithi145 ceph-mon[95801]: osdmap e271: 8 total, 8 up, 8 in 2023-07-12T15:33:03.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-133627-53"}]: dispatch 2023-07-12T15:33:03.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:05.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:33:04 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:33:05.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-133627-53"}]': finished 2023-07-12T15:33:05.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:33:05.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:04 smithi145 ceph-mon[95801]: osdmap e272: 8 total, 8 up, 8 in 2023-07-12T15:33:05.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-37"}]: dispatch 2023-07-12T15:33:05.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:05.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:04 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:05.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:05.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:04 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:05.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-133627-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:05.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:04 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-133627-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:05.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:04 smithi145 ceph-mon[95801]: pgmap v352: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:05.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-133627-53"}]': finished 2023-07-12T15:33:05.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:33:05.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[94241]: osdmap e272: 8 total, 8 up, 8 in 2023-07-12T15:33:05.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-37"}]: dispatch 2023-07-12T15:33:05.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:05.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-133627-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-133627-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[94241]: pgmap v352: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:05.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/704040696' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi069-133627-53"}]': finished 2023-07-12T15:33:05.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:33:05.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[102777]: osdmap e272: 8 total, 8 up, 8 in 2023-07-12T15:33:05.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-37"}]: dispatch 2023-07-12T15:33:05.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:05.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:05.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:05.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:05.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-133627-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:05.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-133627-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:05.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:04 smithi069 ceph-mon[102777]: pgmap v352: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:06 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:06 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:33:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-37"}]': finished 2023-07-12T15:33:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:06 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-133627-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:06 smithi145 ceph-mon[95801]: osdmap e273: 8 total, 8 up, 8 in 2023-07-12T15:33:06.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-133627-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:06.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:06 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-133627-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:06.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:33:06] "GET /metrics HTTP/1.1" 200 225858 "" "Prometheus/2.43.0" 2023-07-12T15:33:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:33:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-37"}]': finished 2023-07-12T15:33:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-133627-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[102777]: osdmap e273: 8 total, 8 up, 8 in 2023-07-12T15:33:06.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-133627-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:06.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-133627-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:06.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:06.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:33:06.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-37"}]': finished 2023-07-12T15:33:06.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi069-133627-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:06.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[94241]: osdmap e273: 8 total, 8 up, 8 in 2023-07-12T15:33:06.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-133627-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:06.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:06 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-133627-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:07 smithi145 ceph-mon[95801]: pgmap v355: 196 pgs: 196 active+clean; 4.6 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:33:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:07 smithi145 ceph-mon[95801]: osdmap e274: 8 total, 8 up, 8 in 2023-07-12T15:33:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:07 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-133627-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-133627-54"}]': finished 2023-07-12T15:33:07.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:07 smithi145 ceph-mon[95801]: osdmap e275: 8 total, 8 up, 8 in 2023-07-12T15:33:07.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:07.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:07 smithi069 ceph-mon[102777]: pgmap v355: 196 pgs: 196 active+clean; 4.6 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:33:07.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:07 smithi069 ceph-mon[102777]: osdmap e274: 8 total, 8 up, 8 in 2023-07-12T15:33:07.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:07 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-133627-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-133627-54"}]': finished 2023-07-12T15:33:07.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:07 smithi069 ceph-mon[102777]: osdmap e275: 8 total, 8 up, 8 in 2023-07-12T15:33:07.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:07.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:07 smithi069 ceph-mon[94241]: pgmap v355: 196 pgs: 196 active+clean; 4.6 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:33:07.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:07 smithi069 ceph-mon[94241]: osdmap e274: 8 total, 8 up, 8 in 2023-07-12T15:33:07.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:07 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi069-133627-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi069-133627-54"}]': finished 2023-07-12T15:33:07.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:07 smithi069 ceph-mon[94241]: osdmap e275: 8 total, 8 up, 8 in 2023-07-12T15:33:07.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-39","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:08 smithi145 ceph-mon[95801]: osdmap e276: 8 total, 8 up, 8 in 2023-07-12T15:33:09.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-39", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:09.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:08 smithi145 ceph-mon[95801]: pgmap v358: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:33:09.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-39","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:09.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:08 smithi069 ceph-mon[102777]: osdmap e276: 8 total, 8 up, 8 in 2023-07-12T15:33:09.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-39", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:09.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:08 smithi069 ceph-mon[102777]: pgmap v358: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:33:09.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-39","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:09.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:08 smithi069 ceph-mon[94241]: osdmap e276: 8 total, 8 up, 8 in 2023-07-12T15:33:09.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-39", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:09.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:08 smithi069 ceph-mon[94241]: pgmap v358: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:33:10.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-39", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:10.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:09 smithi145 ceph-mon[95801]: osdmap e277: 8 total, 8 up, 8 in 2023-07-12T15:33:10.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-39"}]: dispatch 2023-07-12T15:33:10.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:10.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:09 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:10.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-39", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:10.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:09 smithi069 ceph-mon[94241]: osdmap e277: 8 total, 8 up, 8 in 2023-07-12T15:33:10.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-39"}]: dispatch 2023-07-12T15:33:10.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:10.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:09 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:10.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-39", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:10.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:09 smithi069 ceph-mon[102777]: osdmap e277: 8 total, 8 up, 8 in 2023-07-12T15:33:10.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-39"}]: dispatch 2023-07-12T15:33:10.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:10.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:09 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:11.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-39"}]': finished 2023-07-12T15:33:11.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:10 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]': finished 2023-07-12T15:33:11.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:10 smithi145 ceph-mon[95801]: osdmap e278: 8 total, 8 up, 8 in 2023-07-12T15:33:11.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-39", "mode": "writeback"}]: dispatch 2023-07-12T15:33:11.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:11.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:10 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:11.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:10 smithi145 ceph-mon[95801]: pgmap v361: 228 pgs: 228 active+clean; 4.6 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:11.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:10 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:33:11.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-39"}]': finished 2023-07-12T15:33:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]': finished 2023-07-12T15:33:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[102777]: osdmap e278: 8 total, 8 up, 8 in 2023-07-12T15:33:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-39", "mode": "writeback"}]: dispatch 2023-07-12T15:33:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:11.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:11.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[102777]: pgmap v361: 228 pgs: 228 active+clean; 4.6 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:11.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:33:11.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-39"}]': finished 2023-07-12T15:33:11.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi069-133627-54"}]': finished 2023-07-12T15:33:11.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[94241]: osdmap e278: 8 total, 8 up, 8 in 2023-07-12T15:33:11.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-39", "mode": "writeback"}]: dispatch 2023-07-12T15:33:11.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4224937298' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:11.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]: dispatch 2023-07-12T15:33:11.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[94241]: pgmap v361: 228 pgs: 228 active+clean; 4.6 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:11.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:10 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:33:12.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-39", "mode": "writeback"}]': finished 2023-07-12T15:33:12.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:11 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]': finished 2023-07-12T15:33:12.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:11 smithi145 ceph-mon[95801]: osdmap e279: 8 total, 8 up, 8 in 2023-07-12T15:33:12.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:12.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:12.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-133627-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:12.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:12.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-39", "mode": "writeback"}]': finished 2023-07-12T15:33:12.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]': finished 2023-07-12T15:33:12.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[102777]: osdmap e279: 8 total, 8 up, 8 in 2023-07-12T15:33:12.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:12.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:12.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-133627-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:12.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:12.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-39", "mode": "writeback"}]': finished 2023-07-12T15:33:12.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi069-133627-54"}]': finished 2023-07-12T15:33:12.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[94241]: osdmap e279: 8 total, 8 up, 8 in 2023-07-12T15:33:12.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:12.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:12.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-133627-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:12.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:13.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-133627-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:13.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:33:13.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:12 smithi069 ceph-mon[102777]: osdmap e280: 8 total, 8 up, 8 in 2023-07-12T15:33:13.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-39"}]: dispatch 2023-07-12T15:33:13.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-133627-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:13.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:12 smithi069 ceph-mon[102777]: pgmap v364: 228 pgs: 228 active+clean; 4.6 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:13.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-133627-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:13.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:33:13.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:12 smithi069 ceph-mon[94241]: osdmap e280: 8 total, 8 up, 8 in 2023-07-12T15:33:13.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-39"}]: dispatch 2023-07-12T15:33:13.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-133627-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:13.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:12 smithi069 ceph-mon[94241]: pgmap v364: 228 pgs: 228 active+clean; 4.6 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi069-133627-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:33:13.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:12 smithi145 ceph-mon[95801]: osdmap e280: 8 total, 8 up, 8 in 2023-07-12T15:33:13.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-39"}]: dispatch 2023-07-12T15:33:13.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-133627-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:13.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:12 smithi145 ceph-mon[95801]: pgmap v364: 228 pgs: 228 active+clean; 4.6 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:13.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-07-12T15:33:13.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-07-12T15:33:13.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-07-12T15:33:13.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: seed 133790 2023-07-12T15:33:13.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-07-12T15:33:13.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (258 ms) 2023-07-12T15:33:13.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-07-12T15:33:13.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10719 ms) 2023-07-12T15:33:13.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-07-12T15:33:13.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (30 ms) 2023-07-12T15:33:13.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (11007 ms total) 2023-07-12T15:33:13.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2023-07-12T15:33:13.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-07-12T15:33:13.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-07-12T15:33:13.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (6618 ms) 2023-07-12T15:33:13.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-07-12T15:33:13.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8012 ms) 2023-07-12T15:33:13.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-07-12T15:33:13.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10090 ms) 2023-07-12T15:33:13.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-07-12T15:33:13.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [4] 2023-07-12T15:33:13.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [5,4] 2023-07-12T15:33:13.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [6,5,4] 2023-07-12T15:33:13.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-07-12T15:33:13.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting some heads 2023-07-12T15:33:13.869 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 7 2023-07-12T15:33:13.870 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 6 2023-07-12T15:33:13.870 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 5 2023-07-12T15:33:13.870 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 4 2023-07-12T15:33:13.870 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrubs... 2023-07-12T15:33:13.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2023-07-12T15:33:13.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (45906 ms) 2023-07-12T15:33:13.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-07-12T15:33:13.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9980 ms) 2023-07-12T15:33:13.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-07-12T15:33:13.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8051 ms) 2023-07-12T15:33:13.871 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-07-12T15:33:13.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8017 ms) 2023-07-12T15:33:13.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-07-12T15:33:13.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8273 ms) 2023-07-12T15:33:13.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-07-12T15:33:13.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9815 ms) 2023-07-12T15:33:13.872 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-07-12T15:33:13.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8869 ms) 2023-07-12T15:33:13.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-07-12T15:33:13.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10059 ms) 2023-07-12T15:33:13.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-07-12T15:33:13.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13203 ms) 2023-07-12T15:33:13.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-07-12T15:33:13.873 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7772 ms) 2023-07-12T15:33:13.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-07-12T15:33:13.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8030 ms) 2023-07-12T15:33:13.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-07-12T15:33:13.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13074 ms) 2023-07-12T15:33:13.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-07-12T15:33:13.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8121 ms) 2023-07-12T15:33:13.874 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-07-12T15:33:13.875 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (7818 ms) 2023-07-12T15:33:14.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:13 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:33:14.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-39"}]': finished 2023-07-12T15:33:14.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:13 smithi069 ceph-mon[102777]: osdmap e281: 8 total, 8 up, 8 in 2023-07-12T15:33:14.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:33:14.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:13 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:33:14.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-39"}]': finished 2023-07-12T15:33:14.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:13 smithi069 ceph-mon[94241]: osdmap e281: 8 total, 8 up, 8 in 2023-07-12T15:33:14.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:33:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:13 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:33:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-39"}]': finished 2023-07-12T15:33:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:13 smithi145 ceph-mon[95801]: osdmap e281: 8 total, 8 up, 8 in 2023-07-12T15:33:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:33:15.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:33:14 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:33:15.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-133627-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-133627-55"}]': finished 2023-07-12T15:33:15.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:14 smithi145 ceph-mon[95801]: osdmap e282: 8 total, 8 up, 8 in 2023-07-12T15:33:15.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:14 smithi145 ceph-mon[95801]: pgmap v367: 204 pgs: 8 unknown, 196 active+clean; 4.6 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:15.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-133627-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-133627-55"}]': finished 2023-07-12T15:33:15.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:14 smithi069 ceph-mon[94241]: osdmap e282: 8 total, 8 up, 8 in 2023-07-12T15:33:15.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:14 smithi069 ceph-mon[94241]: pgmap v367: 204 pgs: 8 unknown, 196 active+clean; 4.6 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:15.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi069-133627-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi069-133627-55"}]': finished 2023-07-12T15:33:15.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:14 smithi069 ceph-mon[102777]: osdmap e282: 8 total, 8 up, 8 in 2023-07-12T15:33:15.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:14 smithi069 ceph-mon[102777]: pgmap v367: 204 pgs: 8 unknown, 196 active+clean; 4.6 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:16.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:15 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:16.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:15 smithi069 ceph-mon[102777]: osdmap e283: 8 total, 8 up, 8 in 2023-07-12T15:33:16.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:16.149 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:33:16] "GET /metrics HTTP/1.1" 200 228296 "" "Prometheus/2.43.0" 2023-07-12T15:33:16.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:15 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:16.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:15 smithi069 ceph-mon[94241]: osdmap e283: 8 total, 8 up, 8 in 2023-07-12T15:33:16.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:16.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:15 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:15 smithi145 ceph-mon[95801]: osdmap e283: 8 total, 8 up, 8 in 2023-07-12T15:33:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:17.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-41","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[102777]: osdmap e284: 8 total, 8 up, 8 in 2023-07-12T15:33:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-41", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[102777]: pgmap v370: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2023-07-12T15:33:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-133627-55"}]': finished 2023-07-12T15:33:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-41", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[102777]: osdmap e285: 8 total, 8 up, 8 in 2023-07-12T15:33:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:33:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-41","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[94241]: osdmap e284: 8 total, 8 up, 8 in 2023-07-12T15:33:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-41", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[94241]: pgmap v370: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2023-07-12T15:33:17.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-133627-55"}]': finished 2023-07-12T15:33:17.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-41", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:17.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[94241]: osdmap e285: 8 total, 8 up, 8 in 2023-07-12T15:33:17.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:33:17.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:17.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-41","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:16 smithi145 ceph-mon[95801]: osdmap e284: 8 total, 8 up, 8 in 2023-07-12T15:33:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-41", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:16 smithi145 ceph-mon[95801]: pgmap v370: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2023-07-12T15:33:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi069-133627-55"}]': finished 2023-07-12T15:33:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-41", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:17.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:16 smithi145 ceph-mon[95801]: osdmap e285: 8 total, 8 up, 8 in 2023-07-12T15:33:17.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:33:17.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-133627-55"}]: dispatch 2023-07-12T15:33:19.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:33:19.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-133627-55"}]': finished 2023-07-12T15:33:19.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[94241]: osdmap e286: 8 total, 8 up, 8 in 2023-07-12T15:33:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:33:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-133627-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:19.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-133627-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:19.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[94241]: pgmap v373: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2023-07-12T15:33:19.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:33:19.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-133627-55"}]': finished 2023-07-12T15:33:19.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[102777]: osdmap e286: 8 total, 8 up, 8 in 2023-07-12T15:33:19.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:33:19.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:19.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:19.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:19.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:19.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-133627-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:19.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-133627-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:19.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:18 smithi069 ceph-mon[102777]: pgmap v373: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2023-07-12T15:33:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:33:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/922924784' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi069-133627-55"}]': finished 2023-07-12T15:33:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:18 smithi145 ceph-mon[95801]: osdmap e286: 8 total, 8 up, 8 in 2023-07-12T15:33:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:33:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:18 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:19.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:19.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:18 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:19.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-133627-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:19.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:18 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-133627-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:19.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:18 smithi145 ceph-mon[95801]: pgmap v373: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2023-07-12T15:33:20.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:33:20.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:20 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-133627-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:20.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:20 smithi069 ceph-mon[94241]: osdmap e287: 8 total, 8 up, 8 in 2023-07-12T15:33:20.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-133627-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:20.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:33:20.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:20 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-133627-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:20.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:33:20.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:20 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-133627-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:20.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:20 smithi069 ceph-mon[102777]: osdmap e287: 8 total, 8 up, 8 in 2023-07-12T15:33:20.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-133627-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:20.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:33:20.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:20 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-133627-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:20.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:33:20.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:20 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi069-133627-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:20.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:20 smithi145 ceph-mon[95801]: osdmap e287: 8 total, 8 up, 8 in 2023-07-12T15:33:20.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-133627-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:20.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:33:20.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:20 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-133627-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:21.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_type","val": "explicit_object"}]': finished 2023-07-12T15:33:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[102777]: pgmap v376: 228 pgs: 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[102777]: osdmap e288: 8 total, 8 up, 8 in 2023-07-12T15:33:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:33:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-133627-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-133627-56"}]': finished 2023-07-12T15:33:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[102777]: osdmap e289: 8 total, 8 up, 8 in 2023-07-12T15:33:21.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:33:21.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:33:21.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:33:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_type","val": "explicit_object"}]': finished 2023-07-12T15:33:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[94241]: pgmap v376: 228 pgs: 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[94241]: osdmap e288: 8 total, 8 up, 8 in 2023-07-12T15:33:21.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:33:21.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-133627-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-133627-56"}]': finished 2023-07-12T15:33:21.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[94241]: osdmap e289: 8 total, 8 up, 8 in 2023-07-12T15:33:21.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:33:21.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:33:21.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:21 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:33:21.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-41","var": "hit_set_type","val": "explicit_object"}]': finished 2023-07-12T15:33:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:21 smithi145 ceph-mon[95801]: pgmap v376: 228 pgs: 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:21 smithi145 ceph-mon[95801]: osdmap e288: 8 total, 8 up, 8 in 2023-07-12T15:33:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:21 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:33:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:21 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi069-133627-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi069-133627-56"}]': finished 2023-07-12T15:33:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:21 smithi145 ceph-mon[95801]: osdmap e289: 8 total, 8 up, 8 in 2023-07-12T15:33:21.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:21 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:33:21.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:21 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:33:21.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:21 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:33:22.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:22.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-41"}]: dispatch 2023-07-12T15:33:22.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-41"}]': finished 2023-07-12T15:33:22.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:22 smithi069 ceph-mon[94241]: osdmap e290: 8 total, 8 up, 8 in 2023-07-12T15:33:22.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-41"}]: dispatch 2023-07-12T15:33:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-41"}]': finished 2023-07-12T15:33:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:22 smithi069 ceph-mon[102777]: osdmap e290: 8 total, 8 up, 8 in 2023-07-12T15:33:22.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-41"}]: dispatch 2023-07-12T15:33:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-41"}]': finished 2023-07-12T15:33:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:22 smithi145 ceph-mon[95801]: osdmap e290: 8 total, 8 up, 8 in 2023-07-12T15:33:23.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:23 smithi069 ceph-mon[102777]: pgmap v379: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:23 smithi069 ceph-mon[94241]: pgmap v379: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:23.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:23 smithi145 ceph-mon[95801]: pgmap v379: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[94241]: osdmap e291: 8 total, 8 up, 8 in 2023-07-12T15:33:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]': finished 2023-07-12T15:33:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[94241]: osdmap e292: 8 total, 8 up, 8 in 2023-07-12T15:33:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:24.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:24.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:24.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[102777]: osdmap e291: 8 total, 8 up, 8 in 2023-07-12T15:33:24.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:24.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:24.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]': finished 2023-07-12T15:33:24.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[102777]: osdmap e292: 8 total, 8 up, 8 in 2023-07-12T15:33:24.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:24.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:24.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:24 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:24.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:24 smithi145 ceph-mon[95801]: osdmap e291: 8 total, 8 up, 8 in 2023-07-12T15:33:24.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:24.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:24 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:24.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:24 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi069-133627-56"}]': finished 2023-07-12T15:33:24.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:24 smithi145 ceph-mon[95801]: osdmap e292: 8 total, 8 up, 8 in 2023-07-12T15:33:24.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:24.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/663035936' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:24.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:24 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]: dispatch 2023-07-12T15:33:25.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:33:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:33:25.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[94241]: pgmap v382: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:25.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-43","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:25.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]': finished 2023-07-12T15:33:25.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[94241]: osdmap e293: 8 total, 8 up, 8 in 2023-07-12T15:33:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi069-133790-6","var": "pg_num","format": "json"}]: dispatch 2023-07-12T15:33:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-43", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-133627-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-133627-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:25.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[102777]: pgmap v382: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:25.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-43","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:25.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]': finished 2023-07-12T15:33:25.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[102777]: osdmap e293: 8 total, 8 up, 8 in 2023-07-12T15:33:25.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi069-133790-6","var": "pg_num","format": "json"}]: dispatch 2023-07-12T15:33:25.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-43", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:25.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:25.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:25.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:25.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:25.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-133627-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:25.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-133627-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:25 smithi145 ceph-mon[95801]: pgmap v382: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-43","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi069-133627-56"}]': finished 2023-07-12T15:33:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:25 smithi145 ceph-mon[95801]: osdmap e293: 8 total, 8 up, 8 in 2023-07-12T15:33:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi069-133790-6","var": "pg_num","format": "json"}]: dispatch 2023-07-12T15:33:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-43", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-133627-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:25.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-133627-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:26.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:33:26] "GET /metrics HTTP/1.1" 200 228296 "" "Prometheus/2.43.0" 2023-07-12T15:33:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-43", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:26.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-133627-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:26.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[102777]: osdmap e294: 8 total, 8 up, 8 in 2023-07-12T15:33:26.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_count","val": "8"}]: dispatch 2023-07-12T15:33:26.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-133627-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:26.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-133627-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:26.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:26.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-43", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:26.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-133627-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:26.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[94241]: osdmap e294: 8 total, 8 up, 8 in 2023-07-12T15:33:26.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_count","val": "8"}]: dispatch 2023-07-12T15:33:26.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-133627-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:26.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-133627-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:26 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-43", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi069-133627-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:26 smithi145 ceph-mon[95801]: osdmap e294: 8 total, 8 up, 8 in 2023-07-12T15:33:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_count","val": "8"}]: dispatch 2023-07-12T15:33:26.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-133627-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:26.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-133627-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:27.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:27 smithi069 ceph-mon[94241]: pgmap v385: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_count","val": "8"}]': finished 2023-07-12T15:33:27.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:27 smithi069 ceph-mon[102777]: pgmap v385: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:27.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_count","val": "8"}]': finished 2023-07-12T15:33:27.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:27 smithi145 ceph-mon[95801]: pgmap v385: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:27.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_count","val": "8"}]': finished 2023-07-12T15:33:28.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:28 smithi069 ceph-mon[102777]: osdmap e295: 8 total, 8 up, 8 in 2023-07-12T15:33:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:33:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:28 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-133627-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-133627-57"}]': finished 2023-07-12T15:33:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:33:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:28 smithi069 ceph-mon[102777]: osdmap e296: 8 total, 8 up, 8 in 2023-07-12T15:33:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2023-07-12T15:33:28.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:28 smithi069 ceph-mon[94241]: osdmap e295: 8 total, 8 up, 8 in 2023-07-12T15:33:28.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:33:28.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:28 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-133627-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-133627-57"}]': finished 2023-07-12T15:33:28.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:33:28.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:28 smithi069 ceph-mon[94241]: osdmap e296: 8 total, 8 up, 8 in 2023-07-12T15:33:28.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2023-07-12T15:33:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:28 smithi145 ceph-mon[95801]: osdmap e295: 8 total, 8 up, 8 in 2023-07-12T15:33:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:33:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:28 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi069-133627-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi069-133627-57"}]': finished 2023-07-12T15:33:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:33:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:28 smithi145 ceph-mon[95801]: osdmap e296: 8 total, 8 up, 8 in 2023-07-12T15:33:28.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2023-07-12T15:33:29.159 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2023-07-12T15:33:29.160 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: hmm, no HitSet yet 2023-07-12T15:33:29.160 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: ok, hit_set contains 193:602f83fe:::foo:head 2023-07-12T15:33:29.160 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9025 ms) 2023-07-12T15:33:29.160 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-07-12T15:33:29.160 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg_num = 32 2023-07-12T15:33:29.161 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 0 ls 1689176009,0 2023-07-12T15:33:29.161 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 1 ls 1689176009,0 2023-07-12T15:33:29.161 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 2 ls 1689176009,0 2023-07-12T15:33:29.161 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 3 ls 1689176009,0 2023-07-12T15:33:29.161 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 4 ls 1689176009,0 2023-07-12T15:33:29.161 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 5 ls 1689176009,0 2023-07-12T15:33:29.161 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 6 ls 1689176009,0 2023-07-12T15:33:29.162 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 7 ls 1689176009,0 2023-07-12T15:33:29.162 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 8 ls 1689176009,0 2023-07-12T15:33:29.162 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 9 ls 1689176009,0 2023-07-12T15:33:29.162 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 10 ls 1689176009,0 2023-07-12T15:33:29.162 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 11 ls 1689176009,0 2023-07-12T15:33:29.162 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 12 ls 1689176009,0 2023-07-12T15:33:29.162 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 13 ls 1689176009,0 2023-07-12T15:33:29.162 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 14 ls 1689176009,0 2023-07-12T15:33:29.163 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 15 ls 1689176009,0 2023-07-12T15:33:29.163 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 16 ls 1689176009,0 2023-07-12T15:33:29.163 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 17 ls 1689176009,0 2023-07-12T15:33:29.163 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 18 ls 1689176009,0 2023-07-12T15:33:29.163 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 19 ls 1689176009,0 2023-07-12T15:33:29.163 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 20 ls 1689176009,0 2023-07-12T15:33:29.163 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 21 ls 1689176009,0 2023-07-12T15:33:29.164 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 22 ls 1689176009,0 2023-07-12T15:33:29.164 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 23 ls 1689176009,0 2023-07-12T15:33:29.164 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 24 ls 1689176009,0 2023-07-12T15:33:29.164 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 25 ls 1689176009,0 2023-07-12T15:33:29.164 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 26 ls 1689176009,0 2023-07-12T15:33:29.164 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 27 ls 1689176009,0 2023-07-12T15:33:29.164 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 28 ls 1689176009,0 2023-07-12T15:33:29.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 29 ls 1689176009,0 2023-07-12T15:33:29.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 30 ls 1689176009,0 2023-07-12T15:33:29.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 31 ls 1689176009,0 2023-07-12T15:33:29.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg_num = 32 2023-07-12T15:33:29.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:6cac518f:::0:head 2023-07-12T15:33:29.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:02547ec2:::1:head 2023-07-12T15:33:29.165 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:f905c69b:::2:head 2023-07-12T15:33:29.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:cfc208b3:::3:head 2023-07-12T15:33:29.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:d83876eb:::4:head 2023-07-12T15:33:29.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:b29083e3:::5:head 2023-07-12T15:33:29.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:c4fdafeb:::6:head 2023-07-12T15:33:29.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:5c6b0b28:::7:head 2023-07-12T15:33:29.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:bd63b0f1:::8:head 2023-07-12T15:33:29.166 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:e960b815:::9:head 2023-07-12T15:33:29.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:52ea6a34:::10:head 2023-07-12T15:33:29.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:89d3ae78:::11:head 2023-07-12T15:33:29.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:de5d7c5f:::12:head 2023-07-12T15:33:29.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:566253c9:::13:head 2023-07-12T15:33:29.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:62a1935d:::14:head 2023-07-12T15:33:29.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:863748b0:::15:head 2023-07-12T15:33:29.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:3958e169:::16:head 2023-07-12T15:33:29.167 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:4d4dabf9:::17:head 2023-07-12T15:33:29.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:8391935d:::18:head 2023-07-12T15:33:29.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:28883081:::19:head 2023-07-12T15:33:29.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:69259c59:::20:head 2023-07-12T15:33:29.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:4bdb80b7:::21:head 2023-07-12T15:33:29.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:a11c5d71:::22:head 2023-07-12T15:33:29.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:271af37b:::23:head 2023-07-12T15:33:29.168 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:95b121be:::24:head 2023-07-12T15:33:29.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:58d1031b:::25:head 2023-07-12T15:33:29.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:0a050783:::26:head 2023-07-12T15:33:29.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:c709704c:::27:head 2023-07-12T15:33:29.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:cbe56eaf:::28:head 2023-07-12T15:33:29.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:86b4b162:::29:head 2023-07-12T15:33:29.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:70d89383:::30:head 2023-07-12T15:33:29.169 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:dd450c7c:::31:head 2023-07-12T15:33:29.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:6d5729b1:::32:head 2023-07-12T15:33:29.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:c388f3fb:::33:head 2023-07-12T15:33:29.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:56cfea31:::34:head 2023-07-12T15:33:29.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:9dbc1bf7:::35:head 2023-07-12T15:33:29.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:40b74ccd:::36:head 2023-07-12T15:33:29.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:4d5aaf42:::37:head 2023-07-12T15:33:29.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:920f362c:::38:head 2023-07-12T15:33:29.170 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:6cc53222:::39:head 2023-07-12T15:33:29.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:9cad833f:::40:head 2023-07-12T15:33:29.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:1ea84d41:::41:head 2023-07-12T15:33:29.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:c4480ef6:::42:head 2023-07-12T15:33:29.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:a694361e:::43:head 2023-07-12T15:33:29.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:d1bd33e9:::44:head 2023-07-12T15:33:29.171 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:ddc2cd5d:::45:head 2023-07-12T15:33:29.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:2b782207:::46:head 2023-07-12T15:33:29.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:7b187fca:::47:head 2023-07-12T15:33:29.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:90ecdf6f:::48:head 2023-07-12T15:33:29.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:a5ed95fe:::49:head 2023-07-12T15:33:29.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:ea0eaa55:::50:head 2023-07-12T15:33:29.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:f33ef17b:::51:head 2023-07-12T15:33:29.172 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:a0d1b2f6:::52:head 2023-07-12T15:33:29.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:60c5229e:::53:head 2023-07-12T15:33:29.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:edcbc575:::54:head 2023-07-12T15:33:29.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:102cf253:::55:head 2023-07-12T15:33:29.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:efb7fb0b:::56:head 2023-07-12T15:33:29.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:50d0a326:::57:head 2023-07-12T15:33:29.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:d4dc5daf:::58:head 2023-07-12T15:33:29.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:3a130462:::59:head 2023-07-12T15:33:29.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:ec87ed71:::60:head 2023-07-12T15:33:29.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:d5bc9454:::61:head 2023-07-12T15:33:29.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:3ddfe313:::62:head 2023-07-12T15:33:29.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:7c2816b9:::63:head 2023-07-12T15:33:29.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:47e00e4d:::64:head 2023-07-12T15:33:29.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:c6410c18:::65:head 2023-07-12T15:33:29.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:b48ed237:::66:head 2023-07-12T15:33:29.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:cd63ad31:::67:head 2023-07-12T15:33:29.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:b179e92b:::68:head 2023-07-12T15:33:29.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:0d9f741a:::69:head 2023-07-12T15:33:29.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:6d3352ae:::70:head 2023-07-12T15:33:29.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:c6d5c19e:::71:head 2023-07-12T15:33:29.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:bc4729c3:::72:head 2023-07-12T15:33:29.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:77e930b9:::73:head 2023-07-12T15:33:29.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:0abeecfd:::74:head 2023-07-12T15:33:29.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:b7c37e15:::75:head 2023-07-12T15:33:29.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:b6378398:::76:head 2023-07-12T15:33:29.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:02bd68de:::77:head 2023-07-12T15:33:29.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:cc795d2d:::78:head 2023-07-12T15:33:29.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:630d4fea:::79:head 2023-07-12T15:33:29.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:e0d29ef5:::80:head 2023-07-12T15:33:29.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:fd6f13d2:::81:head 2023-07-12T15:33:29.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:606461d5:::82:head 2023-07-12T15:33:29.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:eadbdc43:::83:head 2023-07-12T15:33:29.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:8761d0bb:::84:head 2023-07-12T15:33:29.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:9ef0186f:::85:head 2023-07-12T15:33:29.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:e0d41294:::86:head 2023-07-12T15:33:29.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:961de695:::87:head 2023-07-12T15:33:29.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:1423148f:::88:head 2023-07-12T15:33:29.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:633a8fa2:::89:head 2023-07-12T15:33:29.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:a8653809:::90:head 2023-07-12T15:33:29.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:3dac8b33:::91:head 2023-07-12T15:33:29.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:35aad435:::92:head 2023-07-12T15:33:29.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:f6dcc343:::93:head 2023-07-12T15:33:29.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:dbbdad87:::94:head 2023-07-12T15:33:29.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:1cb48ce0:::95:head 2023-07-12T15:33:29.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:03cd461c:::96:head 2023-07-12T15:33:29.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:17a4ea99:::97:head 2023-07-12T15:33:29.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:9993c9a7:::98:head 2023-07-12T15:33:29.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:6394211c:::99:head 2023-07-12T15:33:29.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:94c7ae57:::100:head 2023-07-12T15:33:29.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:6fdee5bb:::101:head 2023-07-12T15:33:29.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:9a477fd1:::102:head 2023-07-12T15:33:29.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:eb850916:::103:head 2023-07-12T15:33:29.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:affc56b9:::104:head 2023-07-12T15:33:29.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:b42dc814:::105:head 2023-07-12T15:33:29.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:f319f8f0:::106:head 2023-07-12T15:33:29.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:9a40b9de:::107:head 2023-07-12T15:33:29.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:8b524f28:::108:head 2023-07-12T15:33:29.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:e3de589f:::109:head 2023-07-12T15:33:29.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:90f90a5b:::110:head 2023-07-12T15:33:29.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:a7b4f1d7:::111:head 2023-07-12T15:33:29.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:af51766e:::112:head 2023-07-12T15:33:29.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:b6f90bd1:::113:head 2023-07-12T15:33:29.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:e0261208:::114:head 2023-07-12T15:33:29.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:c9569ef7:::115:head 2023-07-12T15:33:29.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:61bebe50:::116:head 2023-07-12T15:33:29.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:fe93412b:::117:head 2023-07-12T15:33:29.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:d3d38bee:::118:head 2023-07-12T15:33:29.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:3100ba0c:::119:head 2023-07-12T15:33:29.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:d0560ada:::120:head 2023-07-12T15:33:29.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:f0ea8b35:::121:head 2023-07-12T15:33:29.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:766f231a:::122:head 2023-07-12T15:33:29.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:a07a2582:::123:head 2023-07-12T15:33:29.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:bd7c6b3a:::124:head 2023-07-12T15:33:29.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:fb2ddaff:::125:head 2023-07-12T15:33:29.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:4408e1fe:::126:head 2023-07-12T15:33:29.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:ee1df7a7:::127:head 2023-07-12T15:33:29.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:c3002909:::128:head 2023-07-12T15:33:29.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:4f48ffa9:::129:head 2023-07-12T15:33:29.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:edf38733:::130:head 2023-07-12T15:33:29.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:c08425c0:::131:head 2023-07-12T15:33:29.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:5f902d98:::132:head 2023-07-12T15:33:29.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:41ea2c93:::133:head 2023-07-12T15:33:29.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:813cee13:::134:head 2023-07-12T15:33:29.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:0131818d:::135:head 2023-07-12T15:33:29.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:26ba5a85:::136:head 2023-07-12T15:33:29.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:381b8a5a:::137:head 2023-07-12T15:33:29.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:28797e47:::138:head 2023-07-12T15:33:29.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:bfca7f22:::139:head 2023-07-12T15:33:29.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:36807075:::140:head 2023-07-12T15:33:29.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:80b03975:::141:head 2023-07-12T15:33:29.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:5c15709b:::142:head 2023-07-12T15:33:29.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:f39ea15e:::143:head 2023-07-12T15:33:29.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:ea992956:::144:head 2023-07-12T15:33:29.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:48887b1c:::145:head 2023-07-12T15:33:29.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:9f24a9dd:::146:head 2023-07-12T15:33:29.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:987f100b:::147:head 2023-07-12T15:33:29.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:d2dd3581:::148:head 2023-07-12T15:33:29.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:7fed1808:::149:head 2023-07-12T15:33:29.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:c80b70e9:::150:head 2023-07-12T15:33:29.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:85ed90f9:::151:head 2023-07-12T15:33:29.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:36428b24:::152:head 2023-07-12T15:33:29.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:d044c34a:::153:head 2023-07-12T15:33:29.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:7c18bf58:::154:head 2023-07-12T15:33:29.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:d1c21232:::155:head 2023-07-12T15:33:29.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:a7a3c575:::156:head 2023-07-12T15:33:29.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:87da0633:::157:head 2023-07-12T15:33:29.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:d5ac3822:::158:head 2023-07-12T15:33:29.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:3f20522d:::159:head 2023-07-12T15:33:29.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:6ca26563:::160:head 2023-07-12T15:33:29.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:532ce135:::161:head 2023-07-12T15:33:29.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:c78863e6:::162:head 2023-07-12T15:33:29.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:29 smithi069 ceph-mon[102777]: pgmap v388: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:29.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:33:29.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2023-07-12T15:33:29.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:29 smithi069 ceph-mon[102777]: osdmap e297: 8 total, 8 up, 8 in 2023-07-12T15:33:29.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:29 smithi069 ceph-mon[94241]: pgmap v388: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:29.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:33:29.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2023-07-12T15:33:29.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:29 smithi069 ceph-mon[94241]: osdmap e297: 8 total, 8 up, 8 in 2023-07-12T15:33:29.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:29 smithi145 ceph-mon[95801]: pgmap v388: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:33:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2023-07-12T15:33:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:29 smithi145 ceph-mon[95801]: osdmap e297: 8 total, 8 up, 8 in 2023-07-12T15:33:30.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi069-133790-43","var": "pg_num","format": "json"}]: dispatch 2023-07-12T15:33:30.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:30.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-43"}]: dispatch 2023-07-12T15:33:30.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-43"}]': finished 2023-07-12T15:33:30.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[94241]: osdmap e298: 8 total, 8 up, 8 in 2023-07-12T15:33:30.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:30.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:30.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi069-133790-43","var": "pg_num","format": "json"}]: dispatch 2023-07-12T15:33:30.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:30.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-43"}]: dispatch 2023-07-12T15:33:30.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-43"}]': finished 2023-07-12T15:33:30.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[102777]: osdmap e298: 8 total, 8 up, 8 in 2023-07-12T15:33:30.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:30.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:30 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:30.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi069-133790-43","var": "pg_num","format": "json"}]: dispatch 2023-07-12T15:33:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-43"}]: dispatch 2023-07-12T15:33:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-43"}]': finished 2023-07-12T15:33:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:30 smithi145 ceph-mon[95801]: osdmap e298: 8 total, 8 up, 8 in 2023-07-12T15:33:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:30 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:31.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:31 smithi069 ceph-mon[102777]: pgmap v391: 228 pgs: 228 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2023-07-12T15:33:31.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:31 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]': finished 2023-07-12T15:33:31.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:31 smithi069 ceph-mon[102777]: osdmap e299: 8 total, 8 up, 8 in 2023-07-12T15:33:31.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:31.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:31 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:31 smithi069 ceph-mon[94241]: pgmap v391: 228 pgs: 228 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2023-07-12T15:33:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:31 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]': finished 2023-07-12T15:33:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:31 smithi069 ceph-mon[94241]: osdmap e299: 8 total, 8 up, 8 in 2023-07-12T15:33:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:31 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:31.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:31 smithi145 ceph-mon[95801]: pgmap v391: 228 pgs: 228 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2023-07-12T15:33:31.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:31 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi069-133627-57"}]': finished 2023-07-12T15:33:31.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:31 smithi145 ceph-mon[95801]: osdmap e299: 8 total, 8 up, 8 in 2023-07-12T15:33:31.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3375494819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:31.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:31 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]: dispatch 2023-07-12T15:33:33.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]': finished 2023-07-12T15:33:33.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[102777]: osdmap e300: 8 total, 8 up, 8 in 2023-07-12T15:33:33.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:33.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:33.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:33.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:33.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:33.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-133627-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:33.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-133627-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:33.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[102777]: pgmap v394: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:33.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]': finished 2023-07-12T15:33:33.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[94241]: osdmap e300: 8 total, 8 up, 8 in 2023-07-12T15:33:33.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:33.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:33.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:33.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:33.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:33.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-133627-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:33.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-133627-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:33.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:32 smithi069 ceph-mon[94241]: pgmap v394: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:33.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:32 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi069-133627-57"}]': finished 2023-07-12T15:33:33.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:32 smithi145 ceph-mon[95801]: osdmap e300: 8 total, 8 up, 8 in 2023-07-12T15:33:33.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:33.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:33.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:32 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:33.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:33.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:32 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:33.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-133627-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:33.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:32 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-133627-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:33.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:32 smithi145 ceph-mon[95801]: pgmap v394: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:34.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-45","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:34.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:33 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-133627-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:34.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:33 smithi069 ceph-mon[102777]: osdmap e301: 8 total, 8 up, 8 in 2023-07-12T15:33:34.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-133627-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:34.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:33 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-133627-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:34.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-45", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:34.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-45","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:34.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:33 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-133627-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:34.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:33 smithi069 ceph-mon[94241]: osdmap e301: 8 total, 8 up, 8 in 2023-07-12T15:33:34.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-133627-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:34.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:33 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-133627-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:34.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-45", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:34.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-45","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:34.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:33 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi069-133627-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:34.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:33 smithi145 ceph-mon[95801]: osdmap e301: 8 total, 8 up, 8 in 2023-07-12T15:33:34.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-133627-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:34.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:33 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-133627-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:34.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-45", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:34.945 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:33:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:33:34.945 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:34 smithi145 ceph-mon[95801]: pgmap v397: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:34.945 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-45", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:35.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:34 smithi069 ceph-mon[102777]: pgmap v397: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-45", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:34 smithi069 ceph-mon[102777]: osdmap e302: 8 total, 8 up, 8 in 2023-07-12T15:33:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_count","val": "3"}]: dispatch 2023-07-12T15:33:35.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:34 smithi069 ceph-mon[94241]: pgmap v397: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:35.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-45", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:35.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:34 smithi069 ceph-mon[94241]: osdmap e302: 8 total, 8 up, 8 in 2023-07-12T15:33:35.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_count","val": "3"}]: dispatch 2023-07-12T15:33:35.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:34 smithi145 ceph-mon[95801]: osdmap e302: 8 total, 8 up, 8 in 2023-07-12T15:33:35.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_count","val": "3"}]: dispatch 2023-07-12T15:33:36.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:33:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:33:36] "GET /metrics HTTP/1.1" 200 230715 "" "Prometheus/2.43.0" 2023-07-12T15:33:36.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:35 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:36.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:35 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-133627-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-133627-58"}]': finished 2023-07-12T15:33:36.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_count","val": "3"}]': finished 2023-07-12T15:33:36.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:35 smithi069 ceph-mon[102777]: osdmap e303: 8 total, 8 up, 8 in 2023-07-12T15:33:36.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_period","val": "3"}]: dispatch 2023-07-12T15:33:36.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:35 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:36.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:35 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-133627-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-133627-58"}]': finished 2023-07-12T15:33:36.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_count","val": "3"}]': finished 2023-07-12T15:33:36.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:35 smithi069 ceph-mon[94241]: osdmap e303: 8 total, 8 up, 8 in 2023-07-12T15:33:36.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_period","val": "3"}]: dispatch 2023-07-12T15:33:36.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:35 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:36.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:35 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi069-133627-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi069-133627-58"}]': finished 2023-07-12T15:33:36.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_count","val": "3"}]': finished 2023-07-12T15:33:36.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:35 smithi145 ceph-mon[95801]: osdmap e303: 8 total, 8 up, 8 in 2023-07-12T15:33:36.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_period","val": "3"}]: dispatch 2023-07-12T15:33:37.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_period","val": "3"}]': finished 2023-07-12T15:33:37.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:36 smithi069 ceph-mon[94241]: osdmap e304: 8 total, 8 up, 8 in 2023-07-12T15:33:37.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:33:37.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:36 smithi069 ceph-mon[94241]: pgmap v400: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:37.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_period","val": "3"}]': finished 2023-07-12T15:33:37.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:36 smithi069 ceph-mon[102777]: osdmap e304: 8 total, 8 up, 8 in 2023-07-12T15:33:37.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:33:37.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:36 smithi069 ceph-mon[102777]: pgmap v400: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:37.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_period","val": "3"}]': finished 2023-07-12T15:33:37.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:36 smithi145 ceph-mon[95801]: osdmap e304: 8 total, 8 up, 8 in 2023-07-12T15:33:37.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:33:37.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:36 smithi145 ceph-mon[95801]: pgmap v400: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:38.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:33:38.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[94241]: osdmap e305: 8 total, 8 up, 8 in 2023-07-12T15:33:38.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:38.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2023-07-12T15:33:38.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:38.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_fpp","val": ".01"}]': finished 2023-07-12T15:33:38.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]': finished 2023-07-12T15:33:38.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[94241]: osdmap e306: 8 total, 8 up, 8 in 2023-07-12T15:33:38.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:38.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:38.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:33:38.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[102777]: osdmap e305: 8 total, 8 up, 8 in 2023-07-12T15:33:38.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:38.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2023-07-12T15:33:38.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:38.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_fpp","val": ".01"}]': finished 2023-07-12T15:33:38.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]': finished 2023-07-12T15:33:38.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[102777]: osdmap e306: 8 total, 8 up, 8 in 2023-07-12T15:33:38.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:38.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:37 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:38.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:33:38.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:37 smithi145 ceph-mon[95801]: osdmap e305: 8 total, 8 up, 8 in 2023-07-12T15:33:38.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:38.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2023-07-12T15:33:38.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:37 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:38.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-45","var": "hit_set_fpp","val": ".01"}]': finished 2023-07-12T15:33:38.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:37 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi069-133627-58"}]': finished 2023-07-12T15:33:38.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:37 smithi145 ceph-mon[95801]: osdmap e306: 8 total, 8 up, 8 in 2023-07-12T15:33:38.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3948920722' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:38.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:37 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]: dispatch 2023-07-12T15:33:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:38 smithi069 ceph-mon[102777]: pgmap v403: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:38 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]': finished 2023-07-12T15:33:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:38 smithi069 ceph-mon[102777]: osdmap e307: 8 total, 8 up, 8 in 2023-07-12T15:33:39.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:38 smithi069 ceph-mon[94241]: pgmap v403: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:39.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:38 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]': finished 2023-07-12T15:33:39.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:38 smithi069 ceph-mon[94241]: osdmap e307: 8 total, 8 up, 8 in 2023-07-12T15:33:39.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:38 smithi145 ceph-mon[95801]: pgmap v403: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:39.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:38 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi069-133627-58"}]': finished 2023-07-12T15:33:39.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:38 smithi145 ceph-mon[95801]: osdmap e307: 8 total, 8 up, 8 in 2023-07-12T15:33:40.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:40.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:40.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:40.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:40.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:40.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:40.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:41.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:40 smithi145 ceph-mon[95801]: osdmap e308: 8 total, 8 up, 8 in 2023-07-12T15:33:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-133627-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:40 smithi145 ceph-mon[95801]: pgmap v406: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:33:41.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:40 smithi069 ceph-mon[94241]: osdmap e308: 8 total, 8 up, 8 in 2023-07-12T15:33:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-133627-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:40 smithi069 ceph-mon[94241]: pgmap v406: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:33:41.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:41.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:40 smithi069 ceph-mon[102777]: osdmap e308: 8 total, 8 up, 8 in 2023-07-12T15:33:41.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-133627-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:41.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:40 smithi069 ceph-mon[102777]: pgmap v406: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:33:42.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:41 smithi145 ceph-mon[95801]: osdmap e309: 8 total, 8 up, 8 in 2023-07-12T15:33:42.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:41 smithi069 ceph-mon[102777]: osdmap e309: 8 total, 8 up, 8 in 2023-07-12T15:33:42.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:41 smithi069 ceph-mon[94241]: osdmap e309: 8 total, 8 up, 8 in 2023-07-12T15:33:43.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-133627-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]': finished 2023-07-12T15:33:43.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:42 smithi145 ceph-mon[95801]: osdmap e310: 8 total, 8 up, 8 in 2023-07-12T15:33:43.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:42 smithi145 ceph-mon[95801]: pgmap v409: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:33:43.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-133627-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]': finished 2023-07-12T15:33:43.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:42 smithi069 ceph-mon[94241]: osdmap e310: 8 total, 8 up, 8 in 2023-07-12T15:33:43.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:42 smithi069 ceph-mon[94241]: pgmap v409: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:33:43.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi069-133627-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]': finished 2023-07-12T15:33:43.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:42 smithi069 ceph-mon[102777]: osdmap e310: 8 total, 8 up, 8 in 2023-07-12T15:33:43.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:42 smithi069 ceph-mon[102777]: pgmap v409: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:33:44.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:43 smithi145 ceph-mon[95801]: osdmap e311: 8 total, 8 up, 8 in 2023-07-12T15:33:44.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:43 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:33:44.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:44 smithi069 ceph-mon[102777]: osdmap e311: 8 total, 8 up, 8 in 2023-07-12T15:33:44.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:44 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:33:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:43 smithi069 ceph-mon[94241]: osdmap e311: 8 total, 8 up, 8 in 2023-07-12T15:33:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:43 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:33:45.011 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:33:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:33:45.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:45 smithi145 ceph-mon[95801]: osdmap e312: 8 total, 8 up, 8 in 2023-07-12T15:33:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:45 smithi145 ceph-mon[95801]: pgmap v412: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:45.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:45 smithi069 ceph-mon[102777]: osdmap e312: 8 total, 8 up, 8 in 2023-07-12T15:33:45.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:45.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:45 smithi069 ceph-mon[102777]: pgmap v412: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:45.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:45 smithi069 ceph-mon[94241]: osdmap e312: 8 total, 8 up, 8 in 2023-07-12T15:33:45.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:45.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:45 smithi069 ceph-mon[94241]: pgmap v412: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:46 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]': finished 2023-07-12T15:33:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:46 smithi145 ceph-mon[95801]: osdmap e313: 8 total, 8 up, 8 in 2023-07-12T15:33:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:46.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:33:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:33:46] "GET /metrics HTTP/1.1" 200 230748 "" "Prometheus/2.43.0" 2023-07-12T15:33:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:46 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]': finished 2023-07-12T15:33:46.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:46 smithi069 ceph-mon[94241]: osdmap e313: 8 total, 8 up, 8 in 2023-07-12T15:33:46.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:46 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:46.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi069-133627-59"}]': finished 2023-07-12T15:33:46.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:46 smithi069 ceph-mon[102777]: osdmap e313: 8 total, 8 up, 8 in 2023-07-12T15:33:46.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-133627-59"}]: dispatch 2023-07-12T15:33:47.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-133627-59"}]': finished 2023-07-12T15:33:47.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:47 smithi145 ceph-mon[95801]: osdmap e314: 8 total, 8 up, 8 in 2023-07-12T15:33:47.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:47 smithi145 ceph-mon[95801]: pgmap v415: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2023-07-12T15:33:47.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-133627-59"}]': finished 2023-07-12T15:33:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:47 smithi069 ceph-mon[102777]: osdmap e314: 8 total, 8 up, 8 in 2023-07-12T15:33:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:47 smithi069 ceph-mon[102777]: pgmap v415: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2023-07-12T15:33:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3260391761' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi069-133627-59"}]': finished 2023-07-12T15:33:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:47 smithi069 ceph-mon[94241]: osdmap e314: 8 total, 8 up, 8 in 2023-07-12T15:33:47.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:47 smithi069 ceph-mon[94241]: pgmap v415: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2023-07-12T15:33:48.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:48 smithi145 ceph-mon[95801]: osdmap e315: 8 total, 8 up, 8 in 2023-07-12T15:33:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2223973572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-133627-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:48.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:48 smithi069 ceph-mon[102777]: osdmap e315: 8 total, 8 up, 8 in 2023-07-12T15:33:48.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2223973572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-133627-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:48.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:48 smithi069 ceph-mon[94241]: osdmap e315: 8 total, 8 up, 8 in 2023-07-12T15:33:48.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2223973572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-133627-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:49.055 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:7462ddf6:::.RoundTripAppendPP (2978 ms) 2023-07-12T15:33:49.056 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-07-12T15:33:49.056 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3239 ms) 2023-07-12T15:33:49.056 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-07-12T15:33:49.056 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2799 ms) 2023-07-12T15:33:49.056 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-07-12T15:33:49.056 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2968 ms) 2023-07-12T15:33:49.056 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-07-12T15:33:49.057 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: setting pool EIO 2023-07-12T15:33:49.057 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: max_success 98, min_failed 99 2023-07-12T15:33:49.057 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4009 ms) 2023-07-12T15:33:49.057 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-07-12T15:33:49.057 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3034 ms) 2023-07-12T15:33:49.057 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (118180 ms total) 2023-07-12T15:33:49.057 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2023-07-12T15:33:49.058 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-07-12T15:33:49.058 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-07-12T15:33:49.058 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3019 ms) 2023-07-12T15:33:49.058 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-07-12T15:33:49.058 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9036 ms) 2023-07-12T15:33:49.058 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-07-12T15:33:49.058 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14912 ms) 2023-07-12T15:33:49.058 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-07-12T15:33:49.059 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3032 ms) 2023-07-12T15:33:49.059 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29999 ms total) 2023-07-12T15:33:49.059 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2023-07-12T15:33:49.059 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-07-12T15:33:49.059 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-07-12T15:33:49.059 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2998 ms) 2023-07-12T15:33:49.059 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2998 ms total) 2023-07-12T15:33:49.059 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2023-07-12T15:33:49.060 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-07-12T15:33:49.060 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-07-12T15:33:49.060 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13975 ms) 2023-07-12T15:33:49.060 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-07-12T15:33:49.060 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6996 ms) 2023-07-12T15:33:49.060 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-07-12T15:33:49.060 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7033 ms) 2023-07-12T15:33:49.061 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-07-12T15:33:49.061 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7048 ms) 2023-07-12T15:33:49.061 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-07-12T15:33:49.061 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3242 ms) 2023-07-12T15:33:49.061 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-07-12T15:33:49.061 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6782 ms) 2023-07-12T15:33:49.061 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-07-12T15:33:49.061 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6900 ms) 2023-07-12T15:33:49.062 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-07-12T15:33:49.062 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7036 ms) 2023-07-12T15:33:49.062 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-07-12T15:33:49.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7043 ms) 2023-07-12T15:33:49.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-07-12T15:33:49.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7031 ms) 2023-07-12T15:33:49.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-07-12T15:33:49.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7025 ms) 2023-07-12T15:33:49.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-07-12T15:33:49.063 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7026 ms) 2023-07-12T15:33:49.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-07-12T15:33:49.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7064 ms) 2023-07-12T15:33:49.064 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2023-07-12T15:33:49.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2223973572' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-133627-60","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:49 smithi145 ceph-mon[95801]: osdmap e316: 8 total, 8 up, 8 in 2023-07-12T15:33:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:49 smithi145 ceph-mon[95801]: pgmap v418: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2023-07-12T15:33:49.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2223973572' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-133627-60","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:49.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:49 smithi069 ceph-mon[102777]: osdmap e316: 8 total, 8 up, 8 in 2023-07-12T15:33:49.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:49 smithi069 ceph-mon[102777]: pgmap v418: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2023-07-12T15:33:49.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2223973572' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi069-133627-60","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:49.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:49 smithi069 ceph-mon[94241]: osdmap e316: 8 total, 8 up, 8 in 2023-07-12T15:33:49.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:49 smithi069 ceph-mon[94241]: pgmap v418: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2023-07-12T15:33:50.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:50 smithi145 ceph-mon[95801]: osdmap e317: 8 total, 8 up, 8 in 2023-07-12T15:33:50.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:50.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:50.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:50.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:50.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-133627-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:50.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-133627-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:50.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:50.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-45"}]: dispatch 2023-07-12T15:33:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[102777]: osdmap e317: 8 total, 8 up, 8 in 2023-07-12T15:33:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:50.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-133627-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:50.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-133627-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:50.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:50.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-45"}]: dispatch 2023-07-12T15:33:50.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[94241]: osdmap e317: 8 total, 8 up, 8 in 2023-07-12T15:33:50.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:50.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:50.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:50.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:50.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-133627-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:50.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-133627-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:50.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:33:50.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-45"}]: dispatch 2023-07-12T15:33:51.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:51 smithi145 ceph-mon[95801]: pgmap v420: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-12T15:33:51.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-133627-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:51.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-45"}]': finished 2023-07-12T15:33:51.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:51 smithi145 ceph-mon[95801]: osdmap e318: 8 total, 8 up, 8 in 2023-07-12T15:33:51.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-133627-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:51.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-133627-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:51.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:51 smithi069 ceph-mon[94241]: pgmap v420: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-12T15:33:51.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-133627-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:51.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-45"}]': finished 2023-07-12T15:33:51.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:51 smithi069 ceph-mon[94241]: osdmap e318: 8 total, 8 up, 8 in 2023-07-12T15:33:51.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-133627-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:51.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-133627-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:51.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:51 smithi069 ceph-mon[102777]: pgmap v420: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-12T15:33:51.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi069-133627-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:51.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-45"}]': finished 2023-07-12T15:33:51.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:51 smithi069 ceph-mon[102777]: osdmap e318: 8 total, 8 up, 8 in 2023-07-12T15:33:51.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-133627-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:51.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-133627-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:52.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:52 smithi145 ceph-mon[95801]: osdmap e319: 8 total, 8 up, 8 in 2023-07-12T15:33:52.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-133627-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-133627-61"}]': finished 2023-07-12T15:33:52.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:52 smithi145 ceph-mon[95801]: osdmap e320: 8 total, 8 up, 8 in 2023-07-12T15:33:52.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:52.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:52 smithi069 ceph-mon[102777]: osdmap e319: 8 total, 8 up, 8 in 2023-07-12T15:33:52.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-133627-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-133627-61"}]': finished 2023-07-12T15:33:52.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:52 smithi069 ceph-mon[102777]: osdmap e320: 8 total, 8 up, 8 in 2023-07-12T15:33:52.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:52.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:52 smithi069 ceph-mon[94241]: osdmap e319: 8 total, 8 up, 8 in 2023-07-12T15:33:52.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi069-133627-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi069-133627-61"}]': finished 2023-07-12T15:33:52.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:52 smithi069 ceph-mon[94241]: osdmap e320: 8 total, 8 up, 8 in 2023-07-12T15:33:52.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:33:53.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:53 smithi145 ceph-mon[95801]: pgmap v423: 196 pgs: 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:53.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-47","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:53.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:53 smithi145 ceph-mon[95801]: osdmap e321: 8 total, 8 up, 8 in 2023-07-12T15:33:53.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:53.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:53 smithi069 ceph-mon[102777]: pgmap v423: 196 pgs: 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:53.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-47","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:53.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:53 smithi069 ceph-mon[102777]: osdmap e321: 8 total, 8 up, 8 in 2023-07-12T15:33:53.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:53.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:53 smithi069 ceph-mon[94241]: pgmap v423: 196 pgs: 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:33:53.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-47","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:33:53.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:53 smithi069 ceph-mon[94241]: osdmap e321: 8 total, 8 up, 8 in 2023-07-12T15:33:53.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:33:55.078 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:33:54 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:33:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:55 smithi145 ceph-mon[95801]: pgmap v426: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:55 smithi145 ceph-mon[95801]: osdmap e322: 8 total, 8 up, 8 in 2023-07-12T15:33:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-47"}]: dispatch 2023-07-12T15:33:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:55 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:55.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:55 smithi069 ceph-mon[94241]: pgmap v426: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:55.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:55.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:55 smithi069 ceph-mon[94241]: osdmap e322: 8 total, 8 up, 8 in 2023-07-12T15:33:55.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-47"}]: dispatch 2023-07-12T15:33:55.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:55.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:55 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:55.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:55 smithi069 ceph-mon[102777]: pgmap v426: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:33:55.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:33:55.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:55 smithi069 ceph-mon[102777]: osdmap e322: 8 total, 8 up, 8 in 2023-07-12T15:33:55.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-47"}]: dispatch 2023-07-12T15:33:55.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:55.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:55 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:56.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:56 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-47"}]': finished 2023-07-12T15:33:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:56 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]': finished 2023-07-12T15:33:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:56 smithi145 ceph-mon[95801]: osdmap e323: 8 total, 8 up, 8 in 2023-07-12T15:33:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-47", "mode": "writeback"}]: dispatch 2023-07-12T15:33:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:56 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:56 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:33:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-47", "mode": "writeback"}]': finished 2023-07-12T15:33:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:56 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]': finished 2023-07-12T15:33:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:56 smithi145 ceph-mon[95801]: osdmap e324: 8 total, 8 up, 8 in 2023-07-12T15:33:56.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:33:56.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:33:56] "GET /metrics HTTP/1.1" 200 230748 "" "Prometheus/2.43.0" 2023-07-12T15:33:56.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:56.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-47"}]': finished 2023-07-12T15:33:56.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]': finished 2023-07-12T15:33:56.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[102777]: osdmap e323: 8 total, 8 up, 8 in 2023-07-12T15:33:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-47", "mode": "writeback"}]: dispatch 2023-07-12T15:33:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:33:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-47", "mode": "writeback"}]': finished 2023-07-12T15:33:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]': finished 2023-07-12T15:33:56.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[102777]: osdmap e324: 8 total, 8 up, 8 in 2023-07-12T15:33:56.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:33:56.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:33:56.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-47"}]': finished 2023-07-12T15:33:56.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi069-133627-61"}]': finished 2023-07-12T15:33:56.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[94241]: osdmap e323: 8 total, 8 up, 8 in 2023-07-12T15:33:56.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-47", "mode": "writeback"}]: dispatch 2023-07-12T15:33:56.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1662792126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:56.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]: dispatch 2023-07-12T15:33:56.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:33:56.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-47", "mode": "writeback"}]': finished 2023-07-12T15:33:56.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi069-133627-61"}]': finished 2023-07-12T15:33:56.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[94241]: osdmap e324: 8 total, 8 up, 8 in 2023-07-12T15:33:56.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:33:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:57 smithi145 ceph-mon[95801]: pgmap v429: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-07-12T15:33:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:57.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:57.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:33:57.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:57.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:57 smithi145 ceph-mon[95801]: osdmap e325: 8 total, 8 up, 8 in 2023-07-12T15:33:57.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:33:57.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-133627-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:57 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-133627-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[94241]: pgmap v429: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-07-12T15:33:57.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:57.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:57.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:33:57.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:57.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[94241]: osdmap e325: 8 total, 8 up, 8 in 2023-07-12T15:33:57.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:33:57.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-133627-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-133627-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[102777]: pgmap v429: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-07-12T15:33:57.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:57.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:33:57.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:33:57.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:33:57.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[102777]: osdmap e325: 8 total, 8 up, 8 in 2023-07-12T15:33:57.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:33:57.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-133627-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:57.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:57 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-133627-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:33:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:59 smithi145 ceph-mon[95801]: pgmap v432: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-07-12T15:33:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:33:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:59 smithi145 ceph-mon[95801]: osdmap e326: 8 total, 8 up, 8 in 2023-07-12T15:33:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:33:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:59 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:33:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:33:59 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:33:59.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:59 smithi069 ceph-mon[94241]: pgmap v432: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-07-12T15:33:59.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:33:59.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:59 smithi069 ceph-mon[94241]: osdmap e326: 8 total, 8 up, 8 in 2023-07-12T15:33:59.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:33:59.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:59 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:33:59.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:33:59 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:33:59.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:59 smithi069 ceph-mon[102777]: pgmap v432: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-07-12T15:33:59.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:33:59.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:59 smithi069 ceph-mon[102777]: osdmap e326: 8 total, 8 up, 8 in 2023-07-12T15:33:59.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:33:59.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:59 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:33:59.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:33:59 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:34:00.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:00 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:34:00.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:00 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-133627-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-133627-62"}]': finished 2023-07-12T15:34:00.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:34:00.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:00 smithi145 ceph-mon[95801]: osdmap e327: 8 total, 8 up, 8 in 2023-07-12T15:34:00.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2023-07-12T15:34:00.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:00 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:34:00.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:00 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-133627-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-133627-62"}]': finished 2023-07-12T15:34:00.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:34:00.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:00 smithi069 ceph-mon[102777]: osdmap e327: 8 total, 8 up, 8 in 2023-07-12T15:34:00.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2023-07-12T15:34:00.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:00 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:34:00.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:00 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi069-133627-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi069-133627-62"}]': finished 2023-07-12T15:34:00.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:34:00.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:00 smithi069 ceph-mon[94241]: osdmap e327: 8 total, 8 up, 8 in 2023-07-12T15:34:00.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2023-07-12T15:34:01.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:01 smithi069 ceph-mon[102777]: pgmap v435: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:01.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2023-07-12T15:34:01.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:01 smithi069 ceph-mon[102777]: osdmap e328: 8 total, 8 up, 8 in 2023-07-12T15:34:01.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2023-07-12T15:34:01.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:01 smithi069 ceph-mon[94241]: pgmap v435: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:01.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2023-07-12T15:34:01.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:01 smithi069 ceph-mon[94241]: osdmap e328: 8 total, 8 up, 8 in 2023-07-12T15:34:01.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2023-07-12T15:34:01.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:01 smithi145 ceph-mon[95801]: pgmap v435: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:01.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2023-07-12T15:34:01.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:01 smithi145 ceph-mon[95801]: osdmap e328: 8 total, 8 up, 8 in 2023-07-12T15:34:01.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2023-07-12T15:34:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2023-07-12T15:34:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[102777]: osdmap e329: 8 total, 8 up, 8 in 2023-07-12T15:34:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2023-07-12T15:34:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:34:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:34:02.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_search_last_n","val": "1"}]': finished 2023-07-12T15:34:02.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]': finished 2023-07-12T15:34:02.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[102777]: osdmap e330: 8 total, 8 up, 8 in 2023-07-12T15:34:02.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2023-07-12T15:34:02.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[94241]: osdmap e329: 8 total, 8 up, 8 in 2023-07-12T15:34:02.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2023-07-12T15:34:02.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:34:02.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:34:02.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_search_last_n","val": "1"}]': finished 2023-07-12T15:34:02.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]': finished 2023-07-12T15:34:02.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:02 smithi069 ceph-mon[94241]: osdmap e330: 8 total, 8 up, 8 in 2023-07-12T15:34:02.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2023-07-12T15:34:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:02 smithi145 ceph-mon[95801]: osdmap e329: 8 total, 8 up, 8 in 2023-07-12T15:34:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2023-07-12T15:34:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:34:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:02 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:34:02.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-47","var": "hit_set_search_last_n","val": "1"}]': finished 2023-07-12T15:34:02.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:02 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi069-133627-62"}]': finished 2023-07-12T15:34:02.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:02 smithi145 ceph-mon[95801]: osdmap e330: 8 total, 8 up, 8 in 2023-07-12T15:34:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:03 smithi069 ceph-mon[102777]: pgmap v438: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:34:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:03 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:34:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:03 smithi069 ceph-mon[94241]: pgmap v438: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:34:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:03 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:34:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:03 smithi145 ceph-mon[95801]: pgmap v438: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1411600473' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:34:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:03 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]: dispatch 2023-07-12T15:34:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:04.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]': finished 2023-07-12T15:34:04.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:34:04.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[102777]: osdmap e331: 8 total, 8 up, 8 in 2023-07-12T15:34:04.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47"}]: dispatch 2023-07-12T15:34:04.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:04.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:04.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-133627-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:04.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]': finished 2023-07-12T15:34:04.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:34:04.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[94241]: osdmap e331: 8 total, 8 up, 8 in 2023-07-12T15:34:04.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47"}]: dispatch 2023-07-12T15:34:04.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:04.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:04.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-133627-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:04.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:04 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi069-133627-62"}]': finished 2023-07-12T15:34:04.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:34:04.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:04 smithi145 ceph-mon[95801]: osdmap e331: 8 total, 8 up, 8 in 2023-07-12T15:34:04.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47"}]: dispatch 2023-07-12T15:34:04.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:04.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:04.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-133627-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:05.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:34:04 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:34:05.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:05 smithi145 ceph-mon[95801]: pgmap v441: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:05.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47"}]': finished 2023-07-12T15:34:05.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-133627-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:34:05.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:05 smithi145 ceph-mon[95801]: osdmap e332: 8 total, 8 up, 8 in 2023-07-12T15:34:05.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-133627-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:05.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:05.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47"}]: dispatch 2023-07-12T15:34:05.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[94241]: pgmap v441: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:05.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47"}]': finished 2023-07-12T15:34:05.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-133627-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:34:05.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[94241]: osdmap e332: 8 total, 8 up, 8 in 2023-07-12T15:34:05.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-133627-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:05.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:05.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47"}]: dispatch 2023-07-12T15:34:05.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[102777]: pgmap v441: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:05.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47"}]': finished 2023-07-12T15:34:05.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi069-133627-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:34:05.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[102777]: osdmap e332: 8 total, 8 up, 8 in 2023-07-12T15:34:05.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-133627-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:05.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:05.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-47"}]: dispatch 2023-07-12T15:34:06.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:34:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:34:06] "GET /metrics HTTP/1.1" 200 228321 "" "Prometheus/2.43.0" 2023-07-12T15:34:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:06 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:06 smithi145 ceph-mon[95801]: osdmap e333: 8 total, 8 up, 8 in 2023-07-12T15:34:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-133627-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-133627-63"}]': finished 2023-07-12T15:34:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:06 smithi145 ceph-mon[95801]: osdmap e334: 8 total, 8 up, 8 in 2023-07-12T15:34:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:06.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:06 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:06.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:06 smithi069 ceph-mon[102777]: osdmap e333: 8 total, 8 up, 8 in 2023-07-12T15:34:06.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-133627-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-133627-63"}]': finished 2023-07-12T15:34:06.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:06 smithi069 ceph-mon[102777]: osdmap e334: 8 total, 8 up, 8 in 2023-07-12T15:34:06.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:06.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:06 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:06.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:06 smithi069 ceph-mon[94241]: osdmap e333: 8 total, 8 up, 8 in 2023-07-12T15:34:06.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi069-133627-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi069-133627-63"}]': finished 2023-07-12T15:34:06.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:06 smithi069 ceph-mon[94241]: osdmap e334: 8 total, 8 up, 8 in 2023-07-12T15:34:06.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:07.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:07 smithi145 ceph-mon[95801]: pgmap v444: 196 pgs: 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-07-12T15:34:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-49","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:07 smithi145 ceph-mon[95801]: osdmap e335: 8 total, 8 up, 8 in 2023-07-12T15:34:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:34:07.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:07 smithi069 ceph-mon[102777]: pgmap v444: 196 pgs: 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-07-12T15:34:07.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-49","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:07.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:07 smithi069 ceph-mon[102777]: osdmap e335: 8 total, 8 up, 8 in 2023-07-12T15:34:07.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:34:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:07 smithi069 ceph-mon[94241]: pgmap v444: 196 pgs: 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-07-12T15:34:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-49","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:07 smithi069 ceph-mon[94241]: osdmap e335: 8 total, 8 up, 8 in 2023-07-12T15:34:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:34:09.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:09 smithi069 ceph-mon[102777]: pgmap v447: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-07-12T15:34:09.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:34:09.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:09 smithi069 ceph-mon[102777]: osdmap e336: 8 total, 8 up, 8 in 2023-07-12T15:34:09.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-49"}]: dispatch 2023-07-12T15:34:09.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:09.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:09 smithi069 ceph-mon[94241]: pgmap v447: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-07-12T15:34:09.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:34:09.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:09 smithi069 ceph-mon[94241]: osdmap e336: 8 total, 8 up, 8 in 2023-07-12T15:34:09.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-49"}]: dispatch 2023-07-12T15:34:09.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:09.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:09 smithi145 ceph-mon[95801]: pgmap v447: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-07-12T15:34:09.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:34:09.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:09 smithi145 ceph-mon[95801]: osdmap e336: 8 total, 8 up, 8 in 2023-07-12T15:34:09.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-49"}]: dispatch 2023-07-12T15:34:09.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:10.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-49"}]': finished 2023-07-12T15:34:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-133627-63"}]': finished 2023-07-12T15:34:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[102777]: osdmap e337: 8 total, 8 up, 8 in 2023-07-12T15:34:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-49", "mode": "readproxy"}]: dispatch 2023-07-12T15:34:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:34:10.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-49", "mode": "readproxy"}]': finished 2023-07-12T15:34:10.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-133627-63"}]': finished 2023-07-12T15:34:10.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[102777]: osdmap e338: 8 total, 8 up, 8 in 2023-07-12T15:34:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-49"}]': finished 2023-07-12T15:34:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-133627-63"}]': finished 2023-07-12T15:34:10.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[94241]: osdmap e337: 8 total, 8 up, 8 in 2023-07-12T15:34:10.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-49", "mode": "readproxy"}]: dispatch 2023-07-12T15:34:10.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:10.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:34:10.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-49", "mode": "readproxy"}]': finished 2023-07-12T15:34:10.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-133627-63"}]': finished 2023-07-12T15:34:10.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:10 smithi069 ceph-mon[94241]: osdmap e338: 8 total, 8 up, 8 in 2023-07-12T15:34:10.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-49"}]': finished 2023-07-12T15:34:10.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi069-133627-63"}]': finished 2023-07-12T15:34:10.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:10 smithi145 ceph-mon[95801]: osdmap e337: 8 total, 8 up, 8 in 2023-07-12T15:34:10.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-49", "mode": "readproxy"}]: dispatch 2023-07-12T15:34:10.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-133627-63"}]: dispatch 2023-07-12T15:34:10.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:10 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:34:10.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-49", "mode": "readproxy"}]': finished 2023-07-12T15:34:10.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1386593567' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi069-133627-63"}]': finished 2023-07-12T15:34:10.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:10 smithi145 ceph-mon[95801]: osdmap e338: 8 total, 8 up, 8 in 2023-07-12T15:34:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:11 smithi069 ceph-mon[102777]: pgmap v450: 228 pgs: 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:34:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-133627-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:11.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:11 smithi069 ceph-mon[94241]: pgmap v450: 228 pgs: 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:34:11.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:11.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:11.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-133627-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:11 smithi145 ceph-mon[95801]: pgmap v450: 228 pgs: 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:34:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-133627-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:12.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-133627-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:34:12.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:12 smithi069 ceph-mon[102777]: osdmap e339: 8 total, 8 up, 8 in 2023-07-12T15:34:12.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-133627-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:12.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-133627-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:34:12.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:12 smithi069 ceph-mon[94241]: osdmap e339: 8 total, 8 up, 8 in 2023-07-12T15:34:12.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-133627-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:12.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi069-133627-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:34:12.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:12 smithi145 ceph-mon[95801]: osdmap e339: 8 total, 8 up, 8 in 2023-07-12T15:34:12.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-133627-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:13.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:13 smithi069 ceph-mon[102777]: pgmap v453: 228 pgs: 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:34:13.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:13 smithi069 ceph-mon[102777]: osdmap e340: 8 total, 8 up, 8 in 2023-07-12T15:34:13.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-133627-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-133627-64"}]': finished 2023-07-12T15:34:13.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:13 smithi069 ceph-mon[102777]: osdmap e341: 8 total, 8 up, 8 in 2023-07-12T15:34:13.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:13 smithi069 ceph-mon[94241]: pgmap v453: 228 pgs: 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:34:13.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:13 smithi069 ceph-mon[94241]: osdmap e340: 8 total, 8 up, 8 in 2023-07-12T15:34:13.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-133627-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-133627-64"}]': finished 2023-07-12T15:34:13.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:13 smithi069 ceph-mon[94241]: osdmap e341: 8 total, 8 up, 8 in 2023-07-12T15:34:13.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:13 smithi145 ceph-mon[95801]: pgmap v453: 228 pgs: 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:34:13.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:13 smithi145 ceph-mon[95801]: osdmap e340: 8 total, 8 up, 8 in 2023-07-12T15:34:13.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi069-133627-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi069-133627-64"}]': finished 2023-07-12T15:34:13.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:13 smithi145 ceph-mon[95801]: osdmap e341: 8 total, 8 up, 8 in 2023-07-12T15:34:14.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:14 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:34:14.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:14 smithi069 ceph-mon[102777]: osdmap e342: 8 total, 8 up, 8 in 2023-07-12T15:34:14.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:14 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:34:14.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:14 smithi069 ceph-mon[94241]: osdmap e342: 8 total, 8 up, 8 in 2023-07-12T15:34:14.741 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:14 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:34:14.741 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:14 smithi145 ceph-mon[95801]: osdmap e342: 8 total, 8 up, 8 in 2023-07-12T15:34:15.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:34:14 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:34:15.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:15 smithi069 ceph-mon[102777]: pgmap v456: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:15.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:15 smithi069 ceph-mon[102777]: osdmap e343: 8 total, 8 up, 8 in 2023-07-12T15:34:15.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:15.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:15 smithi069 ceph-mon[94241]: pgmap v456: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:15.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:15 smithi069 ceph-mon[94241]: osdmap e343: 8 total, 8 up, 8 in 2023-07-12T15:34:15.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:15.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:15 smithi145 ceph-mon[95801]: pgmap v456: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:15.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:15 smithi145 ceph-mon[95801]: osdmap e343: 8 total, 8 up, 8 in 2023-07-12T15:34:15.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:16.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:34:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:34:16] "GET /metrics HTTP/1.1" 200 230739 "" "Prometheus/2.43.0" 2023-07-12T15:34:16.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:16 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:16.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-133627-64"}]': finished 2023-07-12T15:34:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:16 smithi069 ceph-mon[94241]: osdmap e344: 8 total, 8 up, 8 in 2023-07-12T15:34:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:16.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:16 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:16.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-133627-64"}]': finished 2023-07-12T15:34:16.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:16 smithi069 ceph-mon[102777]: osdmap e344: 8 total, 8 up, 8 in 2023-07-12T15:34:16.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:16.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:16 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:16.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi069-133627-64"}]': finished 2023-07-12T15:34:16.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:16 smithi145 ceph-mon[95801]: osdmap e344: 8 total, 8 up, 8 in 2023-07-12T15:34:16.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-133627-64"}]: dispatch 2023-07-12T15:34:17.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:17 smithi069 ceph-mon[94241]: pgmap v459: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-12T15:34:17.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-133627-64"}]': finished 2023-07-12T15:34:17.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:17 smithi069 ceph-mon[94241]: osdmap e345: 8 total, 8 up, 8 in 2023-07-12T15:34:17.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:17.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:17.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-133627-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:17.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:17 smithi069 ceph-mon[102777]: pgmap v459: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-12T15:34:17.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-133627-64"}]': finished 2023-07-12T15:34:17.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:17 smithi069 ceph-mon[102777]: osdmap e345: 8 total, 8 up, 8 in 2023-07-12T15:34:17.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:17.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:17.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-133627-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:17 smithi145 ceph-mon[95801]: pgmap v459: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-12T15:34:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2635932567' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi069-133627-64"}]': finished 2023-07-12T15:34:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:17 smithi145 ceph-mon[95801]: osdmap e345: 8 total, 8 up, 8 in 2023-07-12T15:34:17.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:17.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:17.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-133627-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:19.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:19 smithi069 ceph-mon[94241]: pgmap v462: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-12T15:34:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-133627-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:34:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:19 smithi069 ceph-mon[94241]: osdmap e346: 8 total, 8 up, 8 in 2023-07-12T15:34:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-133627-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:19.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:19 smithi069 ceph-mon[102777]: pgmap v462: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-12T15:34:19.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-133627-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:34:19.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:19 smithi069 ceph-mon[102777]: osdmap e346: 8 total, 8 up, 8 in 2023-07-12T15:34:19.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-133627-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:19.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:19 smithi145 ceph-mon[95801]: pgmap v462: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-12T15:34:19.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi069-133627-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:34:19.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:19 smithi145 ceph-mon[95801]: osdmap e346: 8 total, 8 up, 8 in 2023-07-12T15:34:19.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-133627-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:20.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:20 smithi069 ceph-mon[102777]: osdmap e347: 8 total, 8 up, 8 in 2023-07-12T15:34:20.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:20 smithi069 ceph-mon[94241]: osdmap e347: 8 total, 8 up, 8 in 2023-07-12T15:34:20.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:20 smithi145 ceph-mon[95801]: osdmap e347: 8 total, 8 up, 8 in 2023-07-12T15:34:21.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:21 smithi069 ceph-mon[102777]: pgmap v465: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-133627-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-133627-65"}]': finished 2023-07-12T15:34:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:21 smithi069 ceph-mon[102777]: osdmap e348: 8 total, 8 up, 8 in 2023-07-12T15:34:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:21 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:34:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:21 smithi069 ceph-mon[94241]: pgmap v465: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-133627-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-133627-65"}]': finished 2023-07-12T15:34:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:21 smithi069 ceph-mon[94241]: osdmap e348: 8 total, 8 up, 8 in 2023-07-12T15:34:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:21 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:34:21.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:21 smithi145 ceph-mon[95801]: pgmap v465: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi069-133627-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi069-133627-65"}]': finished 2023-07-12T15:34:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:21 smithi145 ceph-mon[95801]: osdmap e348: 8 total, 8 up, 8 in 2023-07-12T15:34:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:21 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:34:22.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:34:22.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: osdmap e349: 8 total, 8 up, 8 in 2023-07-12T15:34:22.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49"}]: dispatch 2023-07-12T15:34:22.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:34:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:34:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:34:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49"}]': finished 2023-07-12T15:34:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: osdmap e350: 8 total, 8 up, 8 in 2023-07-12T15:34:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:34:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: osdmap e349: 8 total, 8 up, 8 in 2023-07-12T15:34:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49"}]: dispatch 2023-07-12T15:34:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:34:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:34:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:34:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49"}]': finished 2023-07-12T15:34:22.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: osdmap e350: 8 total, 8 up, 8 in 2023-07-12T15:34:22.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:34:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: osdmap e349: 8 total, 8 up, 8 in 2023-07-12T15:34:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49"}]: dispatch 2023-07-12T15:34:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:34:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:34:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:34:22.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49"}]': finished 2023-07-12T15:34:22.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: osdmap e350: 8 total, 8 up, 8 in 2023-07-12T15:34:22.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:23.173 INFO:tasks.workunit.client.0.smithi069.stdout: 163:head 2023-07-12T15:34:23.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:5d165639:::164:head 2023-07-12T15:34:23.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:f43765fc:::165:head 2023-07-12T15:34:23.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:b4c720e9:::166:head 2023-07-12T15:34:23.173 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:e694b040:::167:head 2023-07-12T15:34:23.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:afa38db2:::168:head 2023-07-12T15:34:23.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:77ba9f53:::169:head 2023-07-12T15:34:23.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:87495034:::170:head 2023-07-12T15:34:23.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:7c96bf0e:::171:head 2023-07-12T15:34:23.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:dbe346cc:::172:head 2023-07-12T15:34:23.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:e943ec24:::173:head 2023-07-12T15:34:23.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:f97a9c0c:::174:head 2023-07-12T15:34:23.174 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:6f26e74d:::175:head 2023-07-12T15:34:23.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:4f95e106:::176:head 2023-07-12T15:34:23.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:0e6f2f8f:::177:head 2023-07-12T15:34:23.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:05db05f1:::178:head 2023-07-12T15:34:23.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:38a78d66:::179:head 2023-07-12T15:34:23.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:d095610b:::180:head 2023-07-12T15:34:23.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:a1a9d709:::181:head 2023-07-12T15:34:23.175 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:1e5d39db:::182:head 2023-07-12T15:34:23.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:f7df4fb9:::183:head 2023-07-12T15:34:23.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:03a7f161:::184:head 2023-07-12T15:34:23.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:ba70721e:::185:head 2023-07-12T15:34:23.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:28e5662d:::186:head 2023-07-12T15:34:23.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:973d52de:::187:head 2023-07-12T15:34:23.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:4303eb1c:::188:head 2023-07-12T15:34:23.176 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:b990b48e:::189:head 2023-07-12T15:34:23.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:29b8165b:::190:head 2023-07-12T15:34:23.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:3547f197:::191:head 2023-07-12T15:34:23.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:7e260936:::192:head 2023-07-12T15:34:23.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:1abec7b1:::193:head 2023-07-12T15:34:23.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:10fdda93:::194:head 2023-07-12T15:34:23.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:15817eea:::195:head 2023-07-12T15:34:23.177 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:770bab57:::196:head 2023-07-12T15:34:23.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:ed9e13e7:::197:head 2023-07-12T15:34:23.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:71471a8f:::198:head 2023-07-12T15:34:23.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 195:10fb1d02:::199:head 2023-07-12T15:34:23.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8042 ms) 2023-07-12T15:34:23.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-07-12T15:34:23.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176018,0 2023-07-12T15:34:23.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first is 1689176018 2023-07-12T15:34:23.178 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176018,0 2023-07-12T15:34:23.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176018,0 2023-07-12T15:34:23.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176018,1689176020,1689176021,0 2023-07-12T15:34:23.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176018,1689176020,1689176021,0 2023-07-12T15:34:23.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176018,1689176020,1689176021,0 2023-07-12T15:34:23.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176018,1689176020,1689176021,1689176023,1689176024,0 2023-07-12T15:34:23.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176018,1689176020,1689176021,1689176023,1689176024,0 2023-07-12T15:34:23.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176018,1689176020,1689176021,1689176023,1689176024,0 2023-07-12T15:34:23.179 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176018,1689176020,1689176021,1689176023,1689176024,1689176026,1689176027,0 2023-07-12T15:34:23.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176018,1689176020,1689176021,1689176023,1689176024,1689176026,1689176027,0 2023-07-12T15:34:23.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176018,1689176020,1689176021,1689176023,1689176024,1689176026,1689176027,0 2023-07-12T15:34:23.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176021,1689176023,1689176024,1689176026,1689176027,1689176029,1689176030,0 2023-07-12T15:34:23.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first now 1689176021, trimmed 2023-07-12T15:34:23.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20371 ms) 2023-07-12T15:34:23.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-07-12T15:34:23.180 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: foo0 2023-07-12T15:34:23.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-07-12T15:34:23.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14049 ms) 2023-07-12T15:34:23.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-07-12T15:34:23.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:23 smithi069 ceph-mon[102777]: pgmap v468: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:23.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:23.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49"}]: dispatch 2023-07-12T15:34:23.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-133627-65"}]': finished 2023-07-12T15:34:23.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:23 smithi069 ceph-mon[102777]: osdmap e351: 8 total, 8 up, 8 in 2023-07-12T15:34:23.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:23.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:23 smithi069 ceph-mon[94241]: pgmap v468: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:23.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:23.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49"}]: dispatch 2023-07-12T15:34:23.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-133627-65"}]': finished 2023-07-12T15:34:23.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:23 smithi069 ceph-mon[94241]: osdmap e351: 8 total, 8 up, 8 in 2023-07-12T15:34:23.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:23.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:23 smithi145 ceph-mon[95801]: pgmap v468: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:23.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:23.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-49"}]: dispatch 2023-07-12T15:34:23.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi069-133627-65"}]': finished 2023-07-12T15:34:23.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:23 smithi145 ceph-mon[95801]: osdmap e351: 8 total, 8 up, 8 in 2023-07-12T15:34:23.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-133627-65"}]: dispatch 2023-07-12T15:34:25.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:34:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:34:25.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[102777]: pgmap v471: 196 pgs: 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:25.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-133627-65"}]': finished 2023-07-12T15:34:25.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[102777]: osdmap e352: 8 total, 8 up, 8 in 2023-07-12T15:34:25.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:25.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:25.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:25.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:25.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:25.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-133627-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:25.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-133627-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[94241]: pgmap v471: 196 pgs: 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-133627-65"}]': finished 2023-07-12T15:34:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[94241]: osdmap e352: 8 total, 8 up, 8 in 2023-07-12T15:34:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-133627-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-133627-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:25 smithi145 ceph-mon[95801]: pgmap v471: 196 pgs: 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3069091167' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi069-133627-65"}]': finished 2023-07-12T15:34:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:25 smithi145 ceph-mon[95801]: osdmap e352: 8 total, 8 up, 8 in 2023-07-12T15:34:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-133627-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-133627-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:34:26.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:34:26] "GET /metrics HTTP/1.1" 200 230739 "" "Prometheus/2.43.0" 2023-07-12T15:34:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-51","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-133627-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:34:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[94241]: osdmap e353: 8 total, 8 up, 8 in 2023-07-12T15:34:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-133627-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-133627-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:34:26.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:26.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-51","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:26.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-133627-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:34:26.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[102777]: osdmap e353: 8 total, 8 up, 8 in 2023-07-12T15:34:26.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-133627-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:26.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-133627-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:26.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:34:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:26 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-51","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:26.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi069-133627-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:34:26.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:26 smithi145 ceph-mon[95801]: osdmap e353: 8 total, 8 up, 8 in 2023-07-12T15:34:26.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-133627-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:26.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-133627-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:26.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:34:27.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:27 smithi069 ceph-mon[102777]: pgmap v474: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:27.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:34:27.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:27 smithi069 ceph-mon[102777]: osdmap e354: 8 total, 8 up, 8 in 2023-07-12T15:34:27.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-51"}]: dispatch 2023-07-12T15:34:27.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:27 smithi069 ceph-mon[94241]: pgmap v474: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:27.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:34:27.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:27 smithi069 ceph-mon[94241]: osdmap e354: 8 total, 8 up, 8 in 2023-07-12T15:34:27.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-51"}]: dispatch 2023-07-12T15:34:27.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:27 smithi145 ceph-mon[95801]: pgmap v474: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:27.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:34:27.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:27 smithi145 ceph-mon[95801]: osdmap e354: 8 total, 8 up, 8 in 2023-07-12T15:34:27.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-51"}]: dispatch 2023-07-12T15:34:28.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:28 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-133627-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-133627-66"}]': finished 2023-07-12T15:34:28.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-51"}]': finished 2023-07-12T15:34:28.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:28 smithi069 ceph-mon[94241]: osdmap e355: 8 total, 8 up, 8 in 2023-07-12T15:34:28.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-51", "mode": "writeback"}]: dispatch 2023-07-12T15:34:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:28 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-133627-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-133627-66"}]': finished 2023-07-12T15:34:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-51"}]': finished 2023-07-12T15:34:28.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:28 smithi069 ceph-mon[102777]: osdmap e355: 8 total, 8 up, 8 in 2023-07-12T15:34:28.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-51", "mode": "writeback"}]: dispatch 2023-07-12T15:34:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:28 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi069-133627-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi069-133627-66"}]': finished 2023-07-12T15:34:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-51"}]': finished 2023-07-12T15:34:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:28 smithi145 ceph-mon[95801]: osdmap e355: 8 total, 8 up, 8 in 2023-07-12T15:34:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-51", "mode": "writeback"}]: dispatch 2023-07-12T15:34:29.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:29 smithi069 ceph-mon[102777]: pgmap v477: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:29.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:29 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:34:29.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-51", "mode": "writeback"}]': finished 2023-07-12T15:34:29.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:29 smithi069 ceph-mon[102777]: osdmap e356: 8 total, 8 up, 8 in 2023-07-12T15:34:29.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:34:29.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:34:29.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:29 smithi069 ceph-mon[94241]: pgmap v477: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:29.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:29 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:34:29.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-51", "mode": "writeback"}]': finished 2023-07-12T15:34:29.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:29 smithi069 ceph-mon[94241]: osdmap e356: 8 total, 8 up, 8 in 2023-07-12T15:34:29.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:34:29.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:34:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:29 smithi145 ceph-mon[95801]: pgmap v477: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:34:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:29 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:34:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-51", "mode": "writeback"}]': finished 2023-07-12T15:34:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:29 smithi145 ceph-mon[95801]: osdmap e356: 8 total, 8 up, 8 in 2023-07-12T15:34:29.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:34:29.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:34:30.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:34:30.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:30 smithi069 ceph-mon[102777]: osdmap e357: 8 total, 8 up, 8 in 2023-07-12T15:34:30.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:34:30.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:30.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:30 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:30.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:34:30.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:30 smithi069 ceph-mon[94241]: osdmap e357: 8 total, 8 up, 8 in 2023-07-12T15:34:30.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:34:30.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:30.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:30 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:34:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:30 smithi145 ceph-mon[95801]: osdmap e357: 8 total, 8 up, 8 in 2023-07-12T15:34:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:34:30.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:30.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:30 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:31.233 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3032 ms) 2023-07-12T15:34:31.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-07-12T15:34:31.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7024 ms) 2023-07-12T15:34:31.234 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-07-12T15:34:31.235 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7035 ms) 2023-07-12T15:34:31.235 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-07-12T15:34:31.235 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7005 ms) 2023-07-12T15:34:31.235 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-07-12T15:34:31.235 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7028 ms) 2023-07-12T15:34:31.235 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-07-12T15:34:31.235 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7031 ms) 2023-07-12T15:34:31.236 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-07-12T15:34:31.236 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7050 ms) 2023-07-12T15:34:31.236 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139408 ms total) 2023-07-12T15:34:31.236 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2023-07-12T15:34:31.236 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-07-12T15:34:31.236 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (290585 ms total) 2023-07-12T15:34:31.236 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-07-12T15:34:31.238 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.239 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133638 2023-07-12T15:34:31.239 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133638 2023-07-12T15:34:31.239 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.239 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134090 2023-07-12T15:34:31.239 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 134090 2023-07-12T15:34:31.239 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.239 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133815 2023-07-12T15:34:31.240 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133815 2023-07-12T15:34:31.240 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.240 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133754 2023-07-12T15:34:31.240 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133754 2023-07-12T15:34:31.240 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.240 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133662 2023-07-12T15:34:31.240 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133662 2023-07-12T15:34:31.241 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.241 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133951 2023-07-12T15:34:31.241 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133951 2023-07-12T15:34:31.241 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.241 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133630 2023-07-12T15:34:31.241 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133630 2023-07-12T15:34:31.241 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.241 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133991 2023-07-12T15:34:31.242 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133991 2023-07-12T15:34:31.242 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.242 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133857 2023-07-12T15:34:31.242 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133857 2023-07-12T15:34:31.242 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.242 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134098 2023-07-12T15:34:31.242 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 134098 2023-07-12T15:34:31.243 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.243 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133969 2023-07-12T15:34:31.243 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133969 2023-07-12T15:34:31.243 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.243 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134117 2023-07-12T15:34:31.243 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 134117 2023-07-12T15:34:31.243 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.243 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133614 2023-07-12T15:34:31.244 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133614 2023-07-12T15:34:31.244 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.244 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134107 2023-07-12T15:34:31.245 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 134107 2023-07-12T15:34:31.245 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.245 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=134051 2023-07-12T15:34:31.245 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 134051 2023-07-12T15:34:31.245 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2023-07-12T15:34:31.246 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=133786 2023-07-12T15:34:31.246 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 133786 2023-07-12T15:34:31.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[102777]: pgmap v480: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2023-07-12T15:34:31.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:34:31.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]': finished 2023-07-12T15:34:31.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[102777]: osdmap e358: 8 total, 8 up, 8 in 2023-07-12T15:34:31.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:31.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:31.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:34:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[94241]: pgmap v480: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2023-07-12T15:34:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:34:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]': finished 2023-07-12T15:34:31.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[94241]: osdmap e358: 8 total, 8 up, 8 in 2023-07-12T15:34:31.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:31.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:31.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:34:31.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:31 smithi145 ceph-mon[95801]: pgmap v480: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2023-07-12T15:34:31.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:34:31.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:31 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi069-133627-66"}]': finished 2023-07-12T15:34:31.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:31 smithi145 ceph-mon[95801]: osdmap e358: 8 total, 8 up, 8 in 2023-07-12T15:34:31.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3025077459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:31.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:31 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]: dispatch 2023-07-12T15:34:31.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:34:32.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:32 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:34:32.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:32 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]': finished 2023-07-12T15:34:32.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:34:32.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:32 smithi069 ceph-mon[102777]: osdmap e359: 8 total, 8 up, 8 in 2023-07-12T15:34:32.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2023-07-12T15:34:32.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:32 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:34:32.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:32 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]': finished 2023-07-12T15:34:32.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:34:32.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:32 smithi069 ceph-mon[94241]: osdmap e359: 8 total, 8 up, 8 in 2023-07-12T15:34:32.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2023-07-12T15:34:32.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:32 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:34:32.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:32 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi069-133627-66"}]': finished 2023-07-12T15:34:32.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:34:32.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:32 smithi145 ceph-mon[95801]: osdmap e359: 8 total, 8 up, 8 in 2023-07-12T15:34:32.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2023-07-12T15:34:33.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:33 smithi069 ceph-mon[102777]: pgmap v483: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2023-07-12T15:34:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2023-07-12T15:34:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:33 smithi069 ceph-mon[102777]: osdmap e360: 8 total, 8 up, 8 in 2023-07-12T15:34:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "target_max_objects","val": "1"}]: dispatch 2023-07-12T15:34:33.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:33 smithi069 ceph-mon[94241]: pgmap v483: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2023-07-12T15:34:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2023-07-12T15:34:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:33 smithi069 ceph-mon[94241]: osdmap e360: 8 total, 8 up, 8 in 2023-07-12T15:34:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "target_max_objects","val": "1"}]: dispatch 2023-07-12T15:34:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:33 smithi145 ceph-mon[95801]: pgmap v483: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2023-07-12T15:34:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2023-07-12T15:34:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:33 smithi145 ceph-mon[95801]: osdmap e360: 8 total, 8 up, 8 in 2023-07-12T15:34:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "target_max_objects","val": "1"}]: dispatch 2023-07-12T15:34:34.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "target_max_objects","val": "1"}]': finished 2023-07-12T15:34:34.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:34 smithi145 ceph-mon[95801]: osdmap e361: 8 total, 8 up, 8 in 2023-07-12T15:34:34.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "target_max_objects","val": "1"}]': finished 2023-07-12T15:34:34.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:34 smithi069 ceph-mon[102777]: osdmap e361: 8 total, 8 up, 8 in 2023-07-12T15:34:34.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-51","var": "target_max_objects","val": "1"}]': finished 2023-07-12T15:34:34.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:34 smithi069 ceph-mon[94241]: osdmap e361: 8 total, 8 up, 8 in 2023-07-12T15:34:35.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:34:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:34:35.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:35 smithi145 ceph-mon[95801]: pgmap v486: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:35.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:35 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2023-07-12T15:34:35.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:35 smithi069 ceph-mon[102777]: pgmap v486: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:35.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:35 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2023-07-12T15:34:35.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:35 smithi069 ceph-mon[94241]: pgmap v486: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:35.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:35 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2023-07-12T15:34:36.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:34:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:34:36] "GET /metrics HTTP/1.1" 200 230741 "" "Prometheus/2.43.0" 2023-07-12T15:34:36.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:36 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:36.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:36 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:36.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:36 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:37 smithi145 ceph-mon[95801]: pgmap v487: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:37.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:37 smithi069 ceph-mon[102777]: pgmap v487: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:37.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:37 smithi069 ceph-mon[94241]: pgmap v487: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:39.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:39 smithi145 ceph-mon[95801]: pgmap v488: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:39.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:39 smithi069 ceph-mon[102777]: pgmap v488: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:39.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:39 smithi069 ceph-mon[94241]: pgmap v488: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:41.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:41 smithi145 ceph-mon[95801]: pgmap v489: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:41.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:41 smithi069 ceph-mon[102777]: pgmap v489: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:41.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:41 smithi069 ceph-mon[94241]: pgmap v489: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:42 smithi145 ceph-mon[95801]: pgmap v490: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:43.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:42 smithi069 ceph-mon[102777]: pgmap v490: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:42 smithi069 ceph-mon[94241]: pgmap v490: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:44.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:43 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:43 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:34:44.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:43 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:43 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:34:44.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:44.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:43 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:44.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:43 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:34:45.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:34:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:34:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:34:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:44 smithi145 ceph-mon[95801]: osdmap e362: 8 total, 8 up, 8 in 2023-07-12T15:34:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51"}]: dispatch 2023-07-12T15:34:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:44 smithi145 ceph-mon[95801]: pgmap v492: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:45.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:34:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:44 smithi069 ceph-mon[102777]: osdmap e362: 8 total, 8 up, 8 in 2023-07-12T15:34:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51"}]: dispatch 2023-07-12T15:34:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:44 smithi069 ceph-mon[102777]: pgmap v492: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:34:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:44 smithi069 ceph-mon[94241]: osdmap e362: 8 total, 8 up, 8 in 2023-07-12T15:34:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51"}]: dispatch 2023-07-12T15:34:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:44 smithi069 ceph-mon[94241]: pgmap v492: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:34:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:46 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51"}]': finished 2023-07-12T15:34:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:46 smithi145 ceph-mon[95801]: osdmap e363: 8 total, 8 up, 8 in 2023-07-12T15:34:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:46.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51"}]: dispatch 2023-07-12T15:34:46.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:34:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:34:46] "GET /metrics HTTP/1.1" 200 228425 "" "Prometheus/2.43.0" 2023-07-12T15:34:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:46 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51"}]': finished 2023-07-12T15:34:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:46 smithi069 ceph-mon[94241]: osdmap e363: 8 total, 8 up, 8 in 2023-07-12T15:34:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51"}]: dispatch 2023-07-12T15:34:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:46 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51"}]': finished 2023-07-12T15:34:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:46 smithi069 ceph-mon[102777]: osdmap e363: 8 total, 8 up, 8 in 2023-07-12T15:34:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-51"}]: dispatch 2023-07-12T15:34:47.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:47 smithi145 ceph-mon[95801]: osdmap e364: 8 total, 8 up, 8 in 2023-07-12T15:34:47.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:47 smithi145 ceph-mon[95801]: pgmap v495: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-12T15:34:47.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:47 smithi145 ceph-mon[95801]: osdmap e365: 8 total, 8 up, 8 in 2023-07-12T15:34:47.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:47 smithi069 ceph-mon[102777]: osdmap e364: 8 total, 8 up, 8 in 2023-07-12T15:34:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:47 smithi069 ceph-mon[102777]: pgmap v495: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-12T15:34:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:47 smithi069 ceph-mon[102777]: osdmap e365: 8 total, 8 up, 8 in 2023-07-12T15:34:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:47 smithi069 ceph-mon[94241]: osdmap e364: 8 total, 8 up, 8 in 2023-07-12T15:34:47.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:47 smithi069 ceph-mon[94241]: pgmap v495: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-12T15:34:47.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:47 smithi069 ceph-mon[94241]: osdmap e365: 8 total, 8 up, 8 in 2023-07-12T15:34:47.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:48.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:48 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2023-07-12T15:34:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-53","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:48 smithi145 ceph-mon[95801]: osdmap e366: 8 total, 8 up, 8 in 2023-07-12T15:34:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-53"}]: dispatch 2023-07-12T15:34:48.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:48 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2023-07-12T15:34:48.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-53","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:48.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:48 smithi069 ceph-mon[102777]: osdmap e366: 8 total, 8 up, 8 in 2023-07-12T15:34:48.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:48.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-53"}]: dispatch 2023-07-12T15:34:48.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:48 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2023-07-12T15:34:48.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-53","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:48.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:48 smithi069 ceph-mon[94241]: osdmap e366: 8 total, 8 up, 8 in 2023-07-12T15:34:48.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:48.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-53"}]: dispatch 2023-07-12T15:34:49.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:49 smithi145 ceph-mon[95801]: pgmap v498: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-07-12T15:34:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:49 smithi145 ceph-mon[95801]: osdmap e367: 8 total, 8 up, 8 in 2023-07-12T15:34:49.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:49 smithi069 ceph-mon[102777]: pgmap v498: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-07-12T15:34:49.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:49 smithi069 ceph-mon[102777]: osdmap e367: 8 total, 8 up, 8 in 2023-07-12T15:34:49.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:49 smithi069 ceph-mon[94241]: pgmap v498: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-07-12T15:34:49.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:49 smithi069 ceph-mon[94241]: osdmap e367: 8 total, 8 up, 8 in 2023-07-12T15:34:51.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:50 smithi069 ceph-mon[102777]: osdmap e368: 8 total, 8 up, 8 in 2023-07-12T15:34:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:50 smithi069 ceph-mon[102777]: pgmap v501: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:34:51.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:50 smithi069 ceph-mon[94241]: osdmap e368: 8 total, 8 up, 8 in 2023-07-12T15:34:51.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:51.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:50 smithi069 ceph-mon[94241]: pgmap v501: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:34:51.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:50 smithi145 ceph-mon[95801]: osdmap e368: 8 total, 8 up, 8 in 2023-07-12T15:34:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:50 smithi145 ceph-mon[95801]: pgmap v501: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:34:52.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-55","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:52.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:51 smithi069 ceph-mon[102777]: osdmap e369: 8 total, 8 up, 8 in 2023-07-12T15:34:52.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:34:52.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:52.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-55"}]: dispatch 2023-07-12T15:34:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-55","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:51 smithi069 ceph-mon[94241]: osdmap e369: 8 total, 8 up, 8 in 2023-07-12T15:34:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:34:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-55"}]: dispatch 2023-07-12T15:34:52.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-55","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:51 smithi145 ceph-mon[95801]: osdmap e369: 8 total, 8 up, 8 in 2023-07-12T15:34:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:34:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-55"}]: dispatch 2023-07-12T15:34:53.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:52 smithi069 ceph-mon[102777]: osdmap e370: 8 total, 8 up, 8 in 2023-07-12T15:34:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:52 smithi069 ceph-mon[102777]: pgmap v504: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:34:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:52 smithi069 ceph-mon[94241]: osdmap e370: 8 total, 8 up, 8 in 2023-07-12T15:34:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:52 smithi069 ceph-mon[94241]: pgmap v504: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:34:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:52 smithi145 ceph-mon[95801]: osdmap e370: 8 total, 8 up, 8 in 2023-07-12T15:34:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:52 smithi145 ceph-mon[95801]: pgmap v504: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:34:54.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:53 smithi069 ceph-mon[102777]: osdmap e371: 8 total, 8 up, 8 in 2023-07-12T15:34:54.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:54.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:53 smithi069 ceph-mon[94241]: osdmap e371: 8 total, 8 up, 8 in 2023-07-12T15:34:54.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:53 smithi145 ceph-mon[95801]: osdmap e371: 8 total, 8 up, 8 in 2023-07-12T15:34:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:55.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:34:54 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:34:55.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-57","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:55.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:55 smithi069 ceph-mon[102777]: osdmap e372: 8 total, 8 up, 8 in 2023-07-12T15:34:55.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:55.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-57"}]: dispatch 2023-07-12T15:34:55.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:55 smithi069 ceph-mon[102777]: pgmap v507: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:55.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:55 smithi069 ceph-mon[102777]: osdmap e373: 8 total, 8 up, 8 in 2023-07-12T15:34:55.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-57","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:55.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:55 smithi069 ceph-mon[94241]: osdmap e372: 8 total, 8 up, 8 in 2023-07-12T15:34:55.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:55.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-57"}]: dispatch 2023-07-12T15:34:55.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:55 smithi069 ceph-mon[94241]: pgmap v507: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:55.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:55 smithi069 ceph-mon[94241]: osdmap e373: 8 total, 8 up, 8 in 2023-07-12T15:34:55.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-57","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:55 smithi145 ceph-mon[95801]: osdmap e372: 8 total, 8 up, 8 in 2023-07-12T15:34:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-57"}]: dispatch 2023-07-12T15:34:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:55 smithi145 ceph-mon[95801]: pgmap v507: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:34:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:55 smithi145 ceph-mon[95801]: osdmap e373: 8 total, 8 up, 8 in 2023-07-12T15:34:56.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:34:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:34:56] "GET /metrics HTTP/1.1" 200 228425 "" "Prometheus/2.43.0" 2023-07-12T15:34:56.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:56 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:56 smithi069 ceph-mon[102777]: osdmap e374: 8 total, 8 up, 8 in 2023-07-12T15:34:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:56.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:56 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:56.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:56 smithi069 ceph-mon[94241]: osdmap e374: 8 total, 8 up, 8 in 2023-07-12T15:34:56.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:56.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:56 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:34:56.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:56 smithi145 ceph-mon[95801]: osdmap e374: 8 total, 8 up, 8 in 2023-07-12T15:34:56.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:34:57.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:57 smithi069 ceph-mon[102777]: pgmap v510: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2023-07-12T15:34:57.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-59","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:57.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:57 smithi069 ceph-mon[102777]: osdmap e375: 8 total, 8 up, 8 in 2023-07-12T15:34:57.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:34:57.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:57.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-59"}]: dispatch 2023-07-12T15:34:57.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:57 smithi069 ceph-mon[94241]: pgmap v510: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2023-07-12T15:34:57.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-59","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:57.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:57 smithi069 ceph-mon[94241]: osdmap e375: 8 total, 8 up, 8 in 2023-07-12T15:34:57.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:34:57.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:57.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-59"}]: dispatch 2023-07-12T15:34:57.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:57 smithi145 ceph-mon[95801]: pgmap v510: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2023-07-12T15:34:57.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-59","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:34:57.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:57 smithi145 ceph-mon[95801]: osdmap e375: 8 total, 8 up, 8 in 2023-07-12T15:34:57.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:34:57.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:34:57.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-59"}]: dispatch 2023-07-12T15:34:59.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:58 smithi069 ceph-mon[102777]: osdmap e376: 8 total, 8 up, 8 in 2023-07-12T15:34:59.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:58 smithi069 ceph-mon[102777]: pgmap v513: 196 pgs: 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2023-07-12T15:34:59.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:58 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:59.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:58 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:34:59.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:58 smithi069 ceph-mon[94241]: osdmap e376: 8 total, 8 up, 8 in 2023-07-12T15:34:59.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:58 smithi069 ceph-mon[94241]: pgmap v513: 196 pgs: 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2023-07-12T15:34:59.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:58 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:59.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:58 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:34:59.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:58 smithi145 ceph-mon[95801]: osdmap e376: 8 total, 8 up, 8 in 2023-07-12T15:34:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:58 smithi145 ceph-mon[95801]: pgmap v513: 196 pgs: 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2023-07-12T15:34:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:58 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:34:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:58 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:35:00.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:59 smithi069 ceph-mon[102777]: osdmap e377: 8 total, 8 up, 8 in 2023-07-12T15:35:00.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:34:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:35:00.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:59 smithi069 ceph-mon[94241]: osdmap e377: 8 total, 8 up, 8 in 2023-07-12T15:35:00.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:34:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:35:00.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:59 smithi145 ceph-mon[95801]: osdmap e377: 8 total, 8 up, 8 in 2023-07-12T15:35:00.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:34:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:35:01.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-61","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:35:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:00 smithi069 ceph-mon[102777]: osdmap e378: 8 total, 8 up, 8 in 2023-07-12T15:35:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:35:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:35:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:00 smithi069 ceph-mon[102777]: pgmap v516: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:35:01.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-61","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:35:01.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:00 smithi069 ceph-mon[94241]: osdmap e378: 8 total, 8 up, 8 in 2023-07-12T15:35:01.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:35:01.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:35:01.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:00 smithi069 ceph-mon[94241]: pgmap v516: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:35:01.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-61","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:35:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:00 smithi145 ceph-mon[95801]: osdmap e378: 8 total, 8 up, 8 in 2023-07-12T15:35:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:35:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:35:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:00 smithi145 ceph-mon[95801]: pgmap v516: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:35:02.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:35:02.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:02 smithi069 ceph-mon[94241]: osdmap e379: 8 total, 8 up, 8 in 2023-07-12T15:35:02.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:35:02.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-61"}]: dispatch 2023-07-12T15:35:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:35:02.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:02 smithi069 ceph-mon[102777]: osdmap e379: 8 total, 8 up, 8 in 2023-07-12T15:35:02.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:35:02.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-61"}]: dispatch 2023-07-12T15:35:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:35:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:02 smithi145 ceph-mon[95801]: osdmap e379: 8 total, 8 up, 8 in 2023-07-12T15:35:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:35:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-61"}]: dispatch 2023-07-12T15:35:03.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:03 smithi069 ceph-mon[102777]: osdmap e380: 8 total, 8 up, 8 in 2023-07-12T15:35:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:03 smithi069 ceph-mon[102777]: pgmap v519: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:35:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:03 smithi069 ceph-mon[102777]: osdmap e381: 8 total, 8 up, 8 in 2023-07-12T15:35:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:35:03.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:03 smithi069 ceph-mon[94241]: osdmap e380: 8 total, 8 up, 8 in 2023-07-12T15:35:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:03 smithi069 ceph-mon[94241]: pgmap v519: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:35:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:03 smithi069 ceph-mon[94241]: osdmap e381: 8 total, 8 up, 8 in 2023-07-12T15:35:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:35:03.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:03 smithi145 ceph-mon[95801]: osdmap e380: 8 total, 8 up, 8 in 2023-07-12T15:35:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:03 smithi145 ceph-mon[95801]: pgmap v519: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:35:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:03 smithi145 ceph-mon[95801]: osdmap e381: 8 total, 8 up, 8 in 2023-07-12T15:35:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:35:05.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:35:04 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:35:05.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-63","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:35:05.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:04 smithi145 ceph-mon[95801]: osdmap e382: 8 total, 8 up, 8 in 2023-07-12T15:35:05.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:35:05.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:35:05.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:04 smithi145 ceph-mon[95801]: pgmap v522: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:35:05.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-63","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:35:05.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:04 smithi069 ceph-mon[94241]: osdmap e382: 8 total, 8 up, 8 in 2023-07-12T15:35:05.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:35:05.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:35:05.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:04 smithi069 ceph-mon[94241]: pgmap v522: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:35:05.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-63","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:35:05.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:04 smithi069 ceph-mon[102777]: osdmap e382: 8 total, 8 up, 8 in 2023-07-12T15:35:05.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:35:05.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:35:05.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:04 smithi069 ceph-mon[102777]: pgmap v522: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:35:06.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:35:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:35:06] "GET /metrics HTTP/1.1" 200 225941 "" "Prometheus/2.43.0" 2023-07-12T15:35:06.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:05 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:06.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:35:06.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:05 smithi069 ceph-mon[94241]: osdmap e383: 8 total, 8 up, 8 in 2023-07-12T15:35:06.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:05 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:06.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:35:06.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:05 smithi069 ceph-mon[102777]: osdmap e383: 8 total, 8 up, 8 in 2023-07-12T15:35:06.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:05 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:35:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:05 smithi145 ceph-mon[95801]: osdmap e383: 8 total, 8 up, 8 in 2023-07-12T15:35:07.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:06 smithi069 ceph-mon[102777]: osdmap e384: 8 total, 8 up, 8 in 2023-07-12T15:35:07.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:06 smithi069 ceph-mon[102777]: pgmap v525: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:35:07.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:06 smithi069 ceph-mon[94241]: osdmap e384: 8 total, 8 up, 8 in 2023-07-12T15:35:07.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:06 smithi069 ceph-mon[94241]: pgmap v525: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:35:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:06 smithi145 ceph-mon[95801]: osdmap e384: 8 total, 8 up, 8 in 2023-07-12T15:35:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:06 smithi145 ceph-mon[95801]: pgmap v525: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:35:08.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:07 smithi069 ceph-mon[102777]: osdmap e385: 8 total, 8 up, 8 in 2023-07-12T15:35:08.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:07 smithi069 ceph-mon[94241]: osdmap e385: 8 total, 8 up, 8 in 2023-07-12T15:35:08.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:07 smithi145 ceph-mon[95801]: osdmap e385: 8 total, 8 up, 8 in 2023-07-12T15:35:09.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:08 smithi069 ceph-mon[94241]: osdmap e386: 8 total, 8 up, 8 in 2023-07-12T15:35:09.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:08 smithi069 ceph-mon[94241]: pgmap v528: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:35:09.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:08 smithi069 ceph-mon[102777]: osdmap e386: 8 total, 8 up, 8 in 2023-07-12T15:35:09.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:08 smithi069 ceph-mon[102777]: pgmap v528: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:35:09.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:08 smithi145 ceph-mon[95801]: osdmap e386: 8 total, 8 up, 8 in 2023-07-12T15:35:09.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:08 smithi145 ceph-mon[95801]: pgmap v528: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:35:10.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:09 smithi069 ceph-mon[102777]: osdmap e387: 8 total, 8 up, 8 in 2023-07-12T15:35:10.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:09 smithi069 ceph-mon[94241]: osdmap e387: 8 total, 8 up, 8 in 2023-07-12T15:35:10.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:09 smithi145 ceph-mon[95801]: osdmap e387: 8 total, 8 up, 8 in 2023-07-12T15:35:11.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:10 smithi069 ceph-mon[102777]: pgmap v530: 228 pgs: 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2023-07-12T15:35:11.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:10 smithi069 ceph-mon[94241]: pgmap v530: 228 pgs: 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2023-07-12T15:35:11.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:10 smithi145 ceph-mon[95801]: pgmap v530: 228 pgs: 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2023-07-12T15:35:13.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:12 smithi145 ceph-mon[95801]: osdmap e388: 8 total, 8 up, 8 in 2023-07-12T15:35:13.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:12 smithi145 ceph-mon[95801]: pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2023-07-12T15:35:13.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:12 smithi069 ceph-mon[102777]: osdmap e388: 8 total, 8 up, 8 in 2023-07-12T15:35:13.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:12 smithi069 ceph-mon[102777]: pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2023-07-12T15:35:13.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:12 smithi069 ceph-mon[94241]: osdmap e388: 8 total, 8 up, 8 in 2023-07-12T15:35:13.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:12 smithi069 ceph-mon[94241]: pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2023-07-12T15:35:14.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:35:14.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:35:14.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:35:15.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:14 smithi145 ceph-mon[95801]: pgmap v533: 228 pgs: 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2023-07-12T15:35:15.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:35:14 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:35:15.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:14 smithi069 ceph-mon[102777]: pgmap v533: 228 pgs: 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2023-07-12T15:35:15.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:14 smithi069 ceph-mon[94241]: pgmap v533: 228 pgs: 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2023-07-12T15:35:16.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:15 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:16.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:15 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:16.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:15 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:16.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:35:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:35:16] "GET /metrics HTTP/1.1" 200 228409 "" "Prometheus/2.43.0" 2023-07-12T15:35:17.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:16 smithi145 ceph-mon[95801]: pgmap v534: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2023-07-12T15:35:17.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:16 smithi069 ceph-mon[102777]: pgmap v534: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2023-07-12T15:35:17.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:16 smithi069 ceph-mon[94241]: pgmap v534: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2023-07-12T15:35:19.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:19 smithi069 ceph-mon[102777]: pgmap v535: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-07-12T15:35:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:19 smithi069 ceph-mon[94241]: pgmap v535: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-07-12T15:35:19.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:19 smithi145 ceph-mon[95801]: pgmap v535: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-07-12T15:35:20.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:20 smithi069 ceph-mon[102777]: osdmap e389: 8 total, 8 up, 8 in 2023-07-12T15:35:20.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:20 smithi069 ceph-mon[94241]: osdmap e389: 8 total, 8 up, 8 in 2023-07-12T15:35:20.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:20 smithi145 ceph-mon[95801]: osdmap e389: 8 total, 8 up, 8 in 2023-07-12T15:35:21.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:21 smithi069 ceph-mon[102777]: pgmap v537: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:21 smithi069 ceph-mon[94241]: pgmap v537: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:21 smithi145 ceph-mon[95801]: pgmap v537: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:23.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:22 smithi145 ceph-mon[95801]: pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:23.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:22 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:35:23.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:22 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:35:23.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:22 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:35:23.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:22 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:35:23.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:22 smithi069 ceph-mon[102777]: pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:23.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:22 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:35:23.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:22 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:35:23.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:22 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:35:23.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:22 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:35:23.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:22 smithi069 ceph-mon[94241]: pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:23.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:22 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:35:23.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:22 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:35:23.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:22 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:35:23.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:22 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:35:25.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:35:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:35:25.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:25 smithi069 ceph-mon[102777]: pgmap v539: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:25.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:25 smithi069 ceph-mon[94241]: pgmap v539: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:25.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:25 smithi145 ceph-mon[95801]: pgmap v539: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:26.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:26 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:26 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:26.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:35:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:35:26] "GET /metrics HTTP/1.1" 200 228409 "" "Prometheus/2.43.0" 2023-07-12T15:35:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:26 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:27.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:27 smithi069 ceph-mon[102777]: pgmap v540: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:27.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:27 smithi069 ceph-mon[102777]: osdmap e390: 8 total, 8 up, 8 in 2023-07-12T15:35:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:27 smithi069 ceph-mon[94241]: pgmap v540: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:27 smithi069 ceph-mon[94241]: osdmap e390: 8 total, 8 up, 8 in 2023-07-12T15:35:27.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:27 smithi145 ceph-mon[95801]: pgmap v540: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:27.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:27 smithi145 ceph-mon[95801]: osdmap e390: 8 total, 8 up, 8 in 2023-07-12T15:35:29.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:29 smithi069 ceph-mon[94241]: pgmap v542: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:29.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:35:29.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:29 smithi069 ceph-mon[102777]: pgmap v542: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:29.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:35:29.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:29 smithi145 ceph-mon[95801]: pgmap v542: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:35:30.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:30 smithi069 ceph-mon[94241]: osdmap e391: 8 total, 8 up, 8 in 2023-07-12T15:35:30.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:30 smithi069 ceph-mon[102777]: osdmap e391: 8 total, 8 up, 8 in 2023-07-12T15:35:30.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:30 smithi145 ceph-mon[95801]: osdmap e391: 8 total, 8 up, 8 in 2023-07-12T15:35:31.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:31 smithi069 ceph-mon[102777]: pgmap v544: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2023-07-12T15:35:31.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:31 smithi069 ceph-mon[94241]: pgmap v544: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2023-07-12T15:35:31.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:31 smithi145 ceph-mon[95801]: pgmap v544: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2023-07-12T15:35:33.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:32 smithi145 ceph-mon[95801]: pgmap v545: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2023-07-12T15:35:33.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:32 smithi069 ceph-mon[102777]: pgmap v545: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2023-07-12T15:35:33.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:32 smithi069 ceph-mon[94241]: pgmap v545: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2023-07-12T15:35:35.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:35:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:35:35.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:35 smithi069 ceph-mon[102777]: pgmap v546: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-12T15:35:35.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:35 smithi069 ceph-mon[94241]: pgmap v546: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-12T15:35:35.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:35 smithi145 ceph-mon[95801]: pgmap v546: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2023-07-12T15:35:36.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:36 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:36.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:36 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:36.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:35:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:35:36] "GET /metrics HTTP/1.1" 200 228387 "" "Prometheus/2.43.0" 2023-07-12T15:35:36.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:36 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:37.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:37 smithi069 ceph-mon[102777]: pgmap v547: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:35:37.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:37 smithi069 ceph-mon[102777]: osdmap e392: 8 total, 8 up, 8 in 2023-07-12T15:35:37.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:37 smithi069 ceph-mon[94241]: pgmap v547: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:35:37.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:37 smithi069 ceph-mon[94241]: osdmap e392: 8 total, 8 up, 8 in 2023-07-12T15:35:37.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:37 smithi145 ceph-mon[95801]: pgmap v547: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:35:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:37 smithi145 ceph-mon[95801]: osdmap e392: 8 total, 8 up, 8 in 2023-07-12T15:35:39.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:39 smithi069 ceph-mon[102777]: pgmap v549: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:35:39.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:39 smithi069 ceph-mon[94241]: pgmap v549: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:35:39.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:39 smithi145 ceph-mon[95801]: pgmap v549: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:35:40.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:35:40.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-63"}]: dispatch 2023-07-12T15:35:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:35:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-63"}]: dispatch 2023-07-12T15:35:40.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:35:40.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-63"}]: dispatch 2023-07-12T15:35:41.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:41 smithi069 ceph-mon[102777]: pgmap v550: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:41.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:41 smithi069 ceph-mon[102777]: osdmap e393: 8 total, 8 up, 8 in 2023-07-12T15:35:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:41 smithi069 ceph-mon[94241]: pgmap v550: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:41 smithi069 ceph-mon[94241]: osdmap e393: 8 total, 8 up, 8 in 2023-07-12T15:35:41.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:41 smithi145 ceph-mon[95801]: pgmap v550: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:35:41.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:41 smithi145 ceph-mon[95801]: osdmap e393: 8 total, 8 up, 8 in 2023-07-12T15:35:42.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:42 smithi069 ceph-mon[102777]: osdmap e394: 8 total, 8 up, 8 in 2023-07-12T15:35:42.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:35:42.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-65","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:35:42.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:42 smithi069 ceph-mon[102777]: osdmap e395: 8 total, 8 up, 8 in 2023-07-12T15:35:42.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:35:42.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:42 smithi069 ceph-mon[94241]: osdmap e394: 8 total, 8 up, 8 in 2023-07-12T15:35:42.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:35:42.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-65","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:35:42.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:42 smithi069 ceph-mon[94241]: osdmap e395: 8 total, 8 up, 8 in 2023-07-12T15:35:42.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:35:42.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:42 smithi145 ceph-mon[95801]: osdmap e394: 8 total, 8 up, 8 in 2023-07-12T15:35:42.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:35:42.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-65","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:35:42.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:42 smithi145 ceph-mon[95801]: osdmap e395: 8 total, 8 up, 8 in 2023-07-12T15:35:42.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:35:43.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:43 smithi069 ceph-mon[102777]: pgmap v553: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:35:43.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:35:43.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:35:43.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:43 smithi069 ceph-mon[102777]: osdmap e396: 8 total, 8 up, 8 in 2023-07-12T15:35:43.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:43 smithi069 ceph-mon[94241]: pgmap v553: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:35:43.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:35:43.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:35:43.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:43 smithi069 ceph-mon[94241]: osdmap e396: 8 total, 8 up, 8 in 2023-07-12T15:35:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:43 smithi145 ceph-mon[95801]: pgmap v553: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:35:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:35:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:35:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:43 smithi145 ceph-mon[95801]: osdmap e396: 8 total, 8 up, 8 in 2023-07-12T15:35:44.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:44 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:35:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:44 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:35:44.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:44 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:35:45.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:35:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:35:45.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:45 smithi069 ceph-mon[102777]: pgmap v556: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:35:45.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:45 smithi069 ceph-mon[102777]: osdmap e397: 8 total, 8 up, 8 in 2023-07-12T15:35:45.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:45 smithi069 ceph-mon[94241]: pgmap v556: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:35:45.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:45 smithi069 ceph-mon[94241]: osdmap e397: 8 total, 8 up, 8 in 2023-07-12T15:35:45.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:45 smithi145 ceph-mon[95801]: pgmap v556: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:35:45.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:45 smithi145 ceph-mon[95801]: osdmap e397: 8 total, 8 up, 8 in 2023-07-12T15:35:46.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:46 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:46.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:46 smithi069 ceph-mon[94241]: osdmap e398: 8 total, 8 up, 8 in 2023-07-12T15:35:46.489 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:35:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:35:46] "GET /metrics HTTP/1.1" 200 228344 "" "Prometheus/2.43.0" 2023-07-12T15:35:46.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:46 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:46.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:46 smithi069 ceph-mon[102777]: osdmap e398: 8 total, 8 up, 8 in 2023-07-12T15:35:46.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:46 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:46.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:46 smithi145 ceph-mon[95801]: osdmap e398: 8 total, 8 up, 8 in 2023-07-12T15:35:47.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:47 smithi069 ceph-mon[94241]: pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2023-07-12T15:35:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:47 smithi069 ceph-mon[94241]: osdmap e399: 8 total, 8 up, 8 in 2023-07-12T15:35:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:47 smithi069 ceph-mon[102777]: pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2023-07-12T15:35:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:47 smithi069 ceph-mon[102777]: osdmap e399: 8 total, 8 up, 8 in 2023-07-12T15:35:47.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:47 smithi145 ceph-mon[95801]: pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2023-07-12T15:35:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:47 smithi145 ceph-mon[95801]: osdmap e399: 8 total, 8 up, 8 in 2023-07-12T15:35:49.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:49 smithi069 ceph-mon[102777]: pgmap v561: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2023-07-12T15:35:49.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:49 smithi069 ceph-mon[94241]: pgmap v561: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2023-07-12T15:35:49.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:49 smithi145 ceph-mon[95801]: pgmap v561: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2023-07-12T15:35:51.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:51 smithi069 ceph-mon[102777]: pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-12T15:35:51.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:51 smithi069 ceph-mon[94241]: pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-12T15:35:51.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:51 smithi145 ceph-mon[95801]: pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-12T15:35:53.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:52 smithi145 ceph-mon[95801]: osdmap e400: 8 total, 8 up, 8 in 2023-07-12T15:35:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:52 smithi145 ceph-mon[95801]: pgmap v564: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 297 B/s wr, 3 op/s 2023-07-12T15:35:53.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:52 smithi069 ceph-mon[102777]: osdmap e400: 8 total, 8 up, 8 in 2023-07-12T15:35:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:52 smithi069 ceph-mon[102777]: pgmap v564: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 297 B/s wr, 3 op/s 2023-07-12T15:35:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:52 smithi069 ceph-mon[94241]: osdmap e400: 8 total, 8 up, 8 in 2023-07-12T15:35:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:52 smithi069 ceph-mon[94241]: pgmap v564: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 297 B/s wr, 3 op/s 2023-07-12T15:35:55.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:35:54 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:35:55.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:55 smithi069 ceph-mon[102777]: pgmap v565: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:35:55.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:55 smithi069 ceph-mon[94241]: pgmap v565: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:35:55.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:55 smithi145 ceph-mon[95801]: pgmap v565: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:35:56.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:56 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:56.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:56 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:56.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:35:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:35:56] "GET /metrics HTTP/1.1" 200 228344 "" "Prometheus/2.43.0" 2023-07-12T15:35:56.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:56 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:35:57.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:57 smithi069 ceph-mon[102777]: pgmap v566: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 207 B/s wr, 2 op/s 2023-07-12T15:35:57.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:35:57.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-65"}]: dispatch 2023-07-12T15:35:57.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:57 smithi069 ceph-mon[94241]: pgmap v566: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 207 B/s wr, 2 op/s 2023-07-12T15:35:57.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:35:57.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-65"}]: dispatch 2023-07-12T15:35:57.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:57 smithi145 ceph-mon[95801]: pgmap v566: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 207 B/s wr, 2 op/s 2023-07-12T15:35:57.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:35:57.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-65"}]: dispatch 2023-07-12T15:35:58.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:58 smithi069 ceph-mon[102777]: osdmap e401: 8 total, 8 up, 8 in 2023-07-12T15:35:58.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:58 smithi069 ceph-mon[102777]: osdmap e402: 8 total, 8 up, 8 in 2023-07-12T15:35:58.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:35:58.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:58 smithi069 ceph-mon[94241]: osdmap e401: 8 total, 8 up, 8 in 2023-07-12T15:35:58.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:58 smithi069 ceph-mon[94241]: osdmap e402: 8 total, 8 up, 8 in 2023-07-12T15:35:58.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:35:58.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:58 smithi145 ceph-mon[95801]: osdmap e401: 8 total, 8 up, 8 in 2023-07-12T15:35:58.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:58 smithi145 ceph-mon[95801]: osdmap e402: 8 total, 8 up, 8 in 2023-07-12T15:35:58.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:35:59.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:59 smithi069 ceph-mon[102777]: pgmap v568: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:35:59.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:59 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:35:59.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-67","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:35:59.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:59 smithi069 ceph-mon[102777]: osdmap e403: 8 total, 8 up, 8 in 2023-07-12T15:35:59.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:35:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:35:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:59 smithi069 ceph-mon[94241]: pgmap v568: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:35:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:59 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:35:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-67","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:35:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:59 smithi069 ceph-mon[94241]: osdmap e403: 8 total, 8 up, 8 in 2023-07-12T15:35:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:35:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:35:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:59 smithi145 ceph-mon[95801]: pgmap v568: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:35:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:59 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:35:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-67","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:35:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:59 smithi145 ceph-mon[95801]: osdmap e403: 8 total, 8 up, 8 in 2023-07-12T15:35:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:35:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:01.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:01 smithi069 ceph-mon[102777]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:01.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:01 smithi069 ceph-mon[102777]: osdmap e404: 8 total, 8 up, 8 in 2023-07-12T15:36:01.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:01.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-67"}]: dispatch 2023-07-12T15:36:01.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:01 smithi069 ceph-mon[94241]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:01.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:01 smithi069 ceph-mon[94241]: osdmap e404: 8 total, 8 up, 8 in 2023-07-12T15:36:01.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:01.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-67"}]: dispatch 2023-07-12T15:36:01.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:01 smithi145 ceph-mon[95801]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:01.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:01 smithi145 ceph-mon[95801]: osdmap e404: 8 total, 8 up, 8 in 2023-07-12T15:36:01.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:01.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-67"}]: dispatch 2023-07-12T15:36:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:02 smithi069 ceph-mon[102777]: osdmap e405: 8 total, 8 up, 8 in 2023-07-12T15:36:02.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:02 smithi069 ceph-mon[94241]: osdmap e405: 8 total, 8 up, 8 in 2023-07-12T15:36:02.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:02 smithi145 ceph-mon[95801]: osdmap e405: 8 total, 8 up, 8 in 2023-07-12T15:36:03.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:03 smithi069 ceph-mon[102777]: pgmap v574: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:36:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:03 smithi069 ceph-mon[102777]: osdmap e406: 8 total, 8 up, 8 in 2023-07-12T15:36:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:03.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:03 smithi069 ceph-mon[94241]: pgmap v574: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:36:03.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:03 smithi069 ceph-mon[94241]: osdmap e406: 8 total, 8 up, 8 in 2023-07-12T15:36:03.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:03 smithi145 ceph-mon[95801]: pgmap v574: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:36:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:03 smithi145 ceph-mon[95801]: osdmap e406: 8 total, 8 up, 8 in 2023-07-12T15:36:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:04.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-69","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:04.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:04 smithi069 ceph-mon[102777]: osdmap e407: 8 total, 8 up, 8 in 2023-07-12T15:36:04.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:04.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:04.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-69"}]: dispatch 2023-07-12T15:36:04.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:04 smithi069 ceph-mon[102777]: osdmap e408: 8 total, 8 up, 8 in 2023-07-12T15:36:04.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-69","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:04.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:04 smithi069 ceph-mon[94241]: osdmap e407: 8 total, 8 up, 8 in 2023-07-12T15:36:04.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:04.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:04.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-69"}]: dispatch 2023-07-12T15:36:04.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:04 smithi069 ceph-mon[94241]: osdmap e408: 8 total, 8 up, 8 in 2023-07-12T15:36:04.839 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-69","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:04.840 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:04 smithi145 ceph-mon[95801]: osdmap e407: 8 total, 8 up, 8 in 2023-07-12T15:36:04.840 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:04.840 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:04.840 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-69"}]: dispatch 2023-07-12T15:36:04.841 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:04 smithi145 ceph-mon[95801]: osdmap e408: 8 total, 8 up, 8 in 2023-07-12T15:36:05.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:36:04 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:36:05.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:05 smithi069 ceph-mon[102777]: pgmap v577: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:05.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:05 smithi069 ceph-mon[102777]: osdmap e409: 8 total, 8 up, 8 in 2023-07-12T15:36:05.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:05.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:05 smithi069 ceph-mon[94241]: pgmap v577: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:05.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:05 smithi069 ceph-mon[94241]: osdmap e409: 8 total, 8 up, 8 in 2023-07-12T15:36:05.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:05.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:05 smithi145 ceph-mon[95801]: pgmap v577: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:05.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:05 smithi145 ceph-mon[95801]: osdmap e409: 8 total, 8 up, 8 in 2023-07-12T15:36:05.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:06.438 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:36:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:36:06] "GET /metrics HTTP/1.1" 200 228401 "" "Prometheus/2.43.0" 2023-07-12T15:36:06.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:06 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:06.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-71","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:06.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:06 smithi069 ceph-mon[102777]: osdmap e410: 8 total, 8 up, 8 in 2023-07-12T15:36:06.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:06.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:36:06.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:06 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:06.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-71","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:06.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:06 smithi069 ceph-mon[94241]: osdmap e410: 8 total, 8 up, 8 in 2023-07-12T15:36:06.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:06.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:36:06.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:06 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-71","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:06 smithi145 ceph-mon[95801]: osdmap e410: 8 total, 8 up, 8 in 2023-07-12T15:36:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:36:07.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:07 smithi069 ceph-mon[94241]: pgmap v580: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:36:07.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:36:07.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:07 smithi069 ceph-mon[94241]: osdmap e411: 8 total, 8 up, 8 in 2023-07-12T15:36:07.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:07 smithi069 ceph-mon[102777]: pgmap v580: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:36:07.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:36:07.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:07 smithi069 ceph-mon[102777]: osdmap e411: 8 total, 8 up, 8 in 2023-07-12T15:36:07.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:07 smithi145 ceph-mon[95801]: pgmap v580: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:36:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:36:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:07 smithi145 ceph-mon[95801]: osdmap e411: 8 total, 8 up, 8 in 2023-07-12T15:36:09.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:09 smithi069 ceph-mon[94241]: pgmap v583: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:36:09.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:09 smithi069 ceph-mon[94241]: osdmap e412: 8 total, 8 up, 8 in 2023-07-12T15:36:09.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:09 smithi069 ceph-mon[102777]: pgmap v583: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:36:09.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:09 smithi069 ceph-mon[102777]: osdmap e412: 8 total, 8 up, 8 in 2023-07-12T15:36:09.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:09 smithi145 ceph-mon[95801]: pgmap v583: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:36:09.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:09 smithi145 ceph-mon[95801]: osdmap e412: 8 total, 8 up, 8 in 2023-07-12T15:36:11.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:11 smithi069 ceph-mon[102777]: pgmap v585: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2023-07-12T15:36:11.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:11 smithi069 ceph-mon[94241]: pgmap v585: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2023-07-12T15:36:11.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:11 smithi145 ceph-mon[95801]: pgmap v585: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2023-07-12T15:36:13.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:12 smithi145 ceph-mon[95801]: pgmap v586: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2023-07-12T15:36:13.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:12 smithi069 ceph-mon[102777]: pgmap v586: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2023-07-12T15:36:13.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:12 smithi069 ceph-mon[94241]: pgmap v586: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2023-07-12T15:36:14.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:36:14.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:36:14.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:36:15.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:36:14 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:36:15.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:14 smithi145 ceph-mon[95801]: pgmap v587: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 776 B/s rd, 1.6 KiB/s wr, 3 op/s 2023-07-12T15:36:15.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:14 smithi069 ceph-mon[102777]: pgmap v587: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 776 B/s rd, 1.6 KiB/s wr, 3 op/s 2023-07-12T15:36:15.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:14 smithi069 ceph-mon[94241]: pgmap v587: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 776 B/s rd, 1.6 KiB/s wr, 3 op/s 2023-07-12T15:36:16.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:15 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:16.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:15 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:16.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:15 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:16.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:36:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:36:16] "GET /metrics HTTP/1.1" 200 228459 "" "Prometheus/2.43.0" 2023-07-12T15:36:17.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:16 smithi069 ceph-mon[102777]: pgmap v588: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2023-07-12T15:36:17.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:16 smithi069 ceph-mon[94241]: pgmap v588: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2023-07-12T15:36:17.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:16 smithi145 ceph-mon[95801]: pgmap v588: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2023-07-12T15:36:19.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:19 smithi069 ceph-mon[102777]: pgmap v589: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2023-07-12T15:36:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:19 smithi069 ceph-mon[94241]: pgmap v589: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2023-07-12T15:36:19.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:19 smithi145 ceph-mon[95801]: pgmap v589: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2023-07-12T15:36:21.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:21 smithi069 ceph-mon[102777]: pgmap v590: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2023-07-12T15:36:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:21 smithi069 ceph-mon[94241]: pgmap v590: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2023-07-12T15:36:21.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:21 smithi145 ceph-mon[95801]: pgmap v590: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2023-07-12T15:36:23.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:22 smithi145 ceph-mon[95801]: pgmap v591: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:36:23.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:22 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:36:23.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:22 smithi069 ceph-mon[102777]: pgmap v591: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:36:23.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:22 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:36:23.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:22 smithi069 ceph-mon[94241]: pgmap v591: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:36:23.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:22 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:36:24.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:23 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:36:24.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:23 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:36:24.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:23 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:36:24.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:23 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:36:24.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:23 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:36:24.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:23 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:36:24.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:23 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:36:24.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:23 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:36:24.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:23 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:36:25.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:24 smithi069 ceph-mon[94241]: pgmap v592: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:36:25.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:24 smithi069 ceph-mon[102777]: pgmap v592: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:36:25.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:24 smithi145 ceph-mon[95801]: pgmap v592: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:36:25.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:36:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:36:26.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:25 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:26.148 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:25 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:26.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:25 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:26.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:36:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:36:26] "GET /metrics HTTP/1.1" 200 228459 "" "Prometheus/2.43.0" 2023-07-12T15:36:27.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:26 smithi069 ceph-mon[102777]: pgmap v593: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:36:27.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:26 smithi069 ceph-mon[94241]: pgmap v593: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:36:27.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:26 smithi145 ceph-mon[95801]: pgmap v593: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:36:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:29 smithi069 ceph-mon[94241]: pgmap v594: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:36:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-71"}]: dispatch 2023-07-12T15:36:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:36:29.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:29 smithi069 ceph-mon[102777]: pgmap v594: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:36:29.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:29.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-71"}]: dispatch 2023-07-12T15:36:29.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:36:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:29 smithi145 ceph-mon[95801]: pgmap v594: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:36:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-71"}]: dispatch 2023-07-12T15:36:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:36:30.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:30 smithi069 ceph-mon[102777]: osdmap e413: 8 total, 8 up, 8 in 2023-07-12T15:36:30.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:30 smithi069 ceph-mon[102777]: osdmap e414: 8 total, 8 up, 8 in 2023-07-12T15:36:30.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:30.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:30 smithi069 ceph-mon[94241]: osdmap e413: 8 total, 8 up, 8 in 2023-07-12T15:36:30.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:30 smithi069 ceph-mon[94241]: osdmap e414: 8 total, 8 up, 8 in 2023-07-12T15:36:30.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:30.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:30 smithi145 ceph-mon[95801]: osdmap e413: 8 total, 8 up, 8 in 2023-07-12T15:36:30.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:30 smithi145 ceph-mon[95801]: osdmap e414: 8 total, 8 up, 8 in 2023-07-12T15:36:30.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:31.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:31 smithi069 ceph-mon[102777]: pgmap v596: 196 pgs: 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-73","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:31 smithi069 ceph-mon[102777]: osdmap e415: 8 total, 8 up, 8 in 2023-07-12T15:36:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:31.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:31 smithi069 ceph-mon[94241]: pgmap v596: 196 pgs: 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:31.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-73","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:31.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:31 smithi069 ceph-mon[94241]: osdmap e415: 8 total, 8 up, 8 in 2023-07-12T15:36:31.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:31.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:31 smithi145 ceph-mon[95801]: pgmap v596: 196 pgs: 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-73","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:31 smithi145 ceph-mon[95801]: osdmap e415: 8 total, 8 up, 8 in 2023-07-12T15:36:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:33.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:33 smithi069 ceph-mon[94241]: pgmap v599: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:33.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:33 smithi069 ceph-mon[94241]: osdmap e416: 8 total, 8 up, 8 in 2023-07-12T15:36:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:33 smithi069 ceph-mon[102777]: pgmap v599: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:33 smithi069 ceph-mon[102777]: osdmap e416: 8 total, 8 up, 8 in 2023-07-12T15:36:33.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:33 smithi145 ceph-mon[95801]: pgmap v599: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:33 smithi145 ceph-mon[95801]: osdmap e416: 8 total, 8 up, 8 in 2023-07-12T15:36:34.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:34 smithi069 ceph-mon[94241]: osdmap e417: 8 total, 8 up, 8 in 2023-07-12T15:36:34.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:34 smithi069 ceph-mon[102777]: osdmap e417: 8 total, 8 up, 8 in 2023-07-12T15:36:34.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:34 smithi145 ceph-mon[95801]: osdmap e417: 8 total, 8 up, 8 in 2023-07-12T15:36:35.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:36:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:36:35.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:35 smithi069 ceph-mon[94241]: pgmap v602: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:35.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:35 smithi069 ceph-mon[102777]: pgmap v602: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:35.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:35 smithi145 ceph-mon[95801]: pgmap v602: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:36.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:36 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:36.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:36 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:36.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:36:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:36:36] "GET /metrics HTTP/1.1" 200 228389 "" "Prometheus/2.43.0" 2023-07-12T15:36:36.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:36 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:37.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:37 smithi069 ceph-mon[102777]: pgmap v603: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-07-12T15:36:37.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:37 smithi069 ceph-mon[94241]: pgmap v603: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-07-12T15:36:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:37 smithi145 ceph-mon[95801]: pgmap v603: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-07-12T15:36:39.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:39 smithi069 ceph-mon[102777]: pgmap v604: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2023-07-12T15:36:39.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:39 smithi069 ceph-mon[94241]: pgmap v604: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2023-07-12T15:36:39.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:39 smithi145 ceph-mon[95801]: pgmap v604: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2023-07-12T15:36:41.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:41 smithi069 ceph-mon[102777]: pgmap v605: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2023-07-12T15:36:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:41 smithi069 ceph-mon[94241]: pgmap v605: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2023-07-12T15:36:41.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:41 smithi145 ceph-mon[95801]: pgmap v605: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2023-07-12T15:36:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:42 smithi145 ceph-mon[95801]: pgmap v606: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 720 B/s wr, 3 op/s 2023-07-12T15:36:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:42 smithi069 ceph-mon[102777]: pgmap v606: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 720 B/s wr, 3 op/s 2023-07-12T15:36:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:42 smithi069 ceph-mon[94241]: pgmap v606: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 720 B/s wr, 3 op/s 2023-07-12T15:36:44.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:44.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-73"}]: dispatch 2023-07-12T15:36:44.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:44 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:36:44.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-73"}]: dispatch 2023-07-12T15:36:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:44 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:36:44.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:44.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-73"}]: dispatch 2023-07-12T15:36:44.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:44 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:36:45.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:36:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:36:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:45 smithi145 ceph-mon[95801]: osdmap e418: 8 total, 8 up, 8 in 2023-07-12T15:36:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:45 smithi145 ceph-mon[95801]: pgmap v608: 196 pgs: 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 307 B/s wr, 1 op/s 2023-07-12T15:36:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:45 smithi145 ceph-mon[95801]: osdmap e419: 8 total, 8 up, 8 in 2023-07-12T15:36:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:45.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:45 smithi069 ceph-mon[102777]: osdmap e418: 8 total, 8 up, 8 in 2023-07-12T15:36:45.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:45 smithi069 ceph-mon[102777]: pgmap v608: 196 pgs: 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 307 B/s wr, 1 op/s 2023-07-12T15:36:45.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:45 smithi069 ceph-mon[102777]: osdmap e419: 8 total, 8 up, 8 in 2023-07-12T15:36:45.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:45.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:45 smithi069 ceph-mon[94241]: osdmap e418: 8 total, 8 up, 8 in 2023-07-12T15:36:45.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:45 smithi069 ceph-mon[94241]: pgmap v608: 196 pgs: 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 307 B/s wr, 1 op/s 2023-07-12T15:36:45.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:45 smithi069 ceph-mon[94241]: osdmap e419: 8 total, 8 up, 8 in 2023-07-12T15:36:45.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:46.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:36:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:36:46] "GET /metrics HTTP/1.1" 200 228397 "" "Prometheus/2.43.0" 2023-07-12T15:36:46.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:46 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:46.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-75","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:46.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:46 smithi069 ceph-mon[102777]: osdmap e420: 8 total, 8 up, 8 in 2023-07-12T15:36:46.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:46.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:46 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:46.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-75","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:46.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:46 smithi069 ceph-mon[94241]: osdmap e420: 8 total, 8 up, 8 in 2023-07-12T15:36:46.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:46.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:46 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:46.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-75","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:46.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:46 smithi145 ceph-mon[95801]: osdmap e420: 8 total, 8 up, 8 in 2023-07-12T15:36:46.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:47 smithi069 ceph-mon[102777]: pgmap v611: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:47 smithi069 ceph-mon[102777]: osdmap e421: 8 total, 8 up, 8 in 2023-07-12T15:36:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:47 smithi069 ceph-mon[94241]: pgmap v611: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:47 smithi069 ceph-mon[94241]: osdmap e421: 8 total, 8 up, 8 in 2023-07-12T15:36:47.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:47 smithi145 ceph-mon[95801]: pgmap v611: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:47 smithi145 ceph-mon[95801]: osdmap e421: 8 total, 8 up, 8 in 2023-07-12T15:36:48.235 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:36:47 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-0[107243]: 2023-07-12T15:36:47.892+0000 7f5ea33ef700 -1 snap_mapper.add_oid found existing snaps mapped on 62:9168ac96:test-rados-api-smithi069-133790-76::foo:28, removing 2023-07-12T15:36:48.236 INFO:journalctl@ceph.osd.2.smithi069.stdout:Jul 12 15:36:47 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-2[116331]: 2023-07-12T15:36:47.892+0000 7f00b3fbe700 -1 snap_mapper.add_oid found existing snaps mapped on 62:9168ac96:test-rados-api-smithi069-133790-76::foo:28, removing 2023-07-12T15:36:48.350 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:36:47 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:36:47.892+0000 7f55c1685700 -1 snap_mapper.add_oid found existing snaps mapped on 62:9168ac96:test-rados-api-smithi069-133790-76::foo:28, removing 2023-07-12T15:36:49.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:48 smithi069 ceph-mon[94241]: osdmap e422: 8 total, 8 up, 8 in 2023-07-12T15:36:49.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:49.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-75"}]: dispatch 2023-07-12T15:36:49.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:48 smithi069 ceph-mon[94241]: pgmap v614: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:36:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:48 smithi069 ceph-mon[102777]: osdmap e422: 8 total, 8 up, 8 in 2023-07-12T15:36:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-75"}]: dispatch 2023-07-12T15:36:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:48 smithi069 ceph-mon[102777]: pgmap v614: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:36:49.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:48 smithi145 ceph-mon[95801]: osdmap e422: 8 total, 8 up, 8 in 2023-07-12T15:36:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-75"}]: dispatch 2023-07-12T15:36:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:48 smithi145 ceph-mon[95801]: pgmap v614: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:36:50.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:49 smithi069 ceph-mon[102777]: osdmap e423: 8 total, 8 up, 8 in 2023-07-12T15:36:50.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:49 smithi069 ceph-mon[94241]: osdmap e423: 8 total, 8 up, 8 in 2023-07-12T15:36:50.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:49 smithi145 ceph-mon[95801]: osdmap e423: 8 total, 8 up, 8 in 2023-07-12T15:36:51.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:50 smithi069 ceph-mon[94241]: osdmap e424: 8 total, 8 up, 8 in 2023-07-12T15:36:51.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:51.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:50 smithi069 ceph-mon[94241]: pgmap v617: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2023-07-12T15:36:51.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-77","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:51.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:50 smithi069 ceph-mon[94241]: osdmap e425: 8 total, 8 up, 8 in 2023-07-12T15:36:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:50 smithi069 ceph-mon[102777]: osdmap e424: 8 total, 8 up, 8 in 2023-07-12T15:36:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:50 smithi069 ceph-mon[102777]: pgmap v617: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2023-07-12T15:36:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-77","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:50 smithi069 ceph-mon[102777]: osdmap e425: 8 total, 8 up, 8 in 2023-07-12T15:36:51.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:50 smithi145 ceph-mon[95801]: osdmap e424: 8 total, 8 up, 8 in 2023-07-12T15:36:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:50 smithi145 ceph-mon[95801]: pgmap v617: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2023-07-12T15:36:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-77","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:50 smithi145 ceph-mon[95801]: osdmap e425: 8 total, 8 up, 8 in 2023-07-12T15:36:52.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:52.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:51 smithi069 ceph-mon[94241]: osdmap e426: 8 total, 8 up, 8 in 2023-07-12T15:36:52.236 INFO:journalctl@ceph.osd.3.smithi069.stdout:Jul 12 15:36:51 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-3[120895]: 2023-07-12T15:36:51.886+0000 7f9a06da9700 -1 snap_mapper.add_oid found existing snaps mapped on 62:e97c6614:test-rados-api-smithi069-133790-78::foo:29, removing 2023-07-12T15:36:52.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:52.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:51 smithi069 ceph-mon[102777]: osdmap e426: 8 total, 8 up, 8 in 2023-07-12T15:36:52.350 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:36:51 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:36:51.886+0000 7f39ee8dd700 -1 snap_mapper.add_oid found existing snaps mapped on 62:e97c6614:test-rados-api-smithi069-133790-78::foo:29, removing 2023-07-12T15:36:52.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:36:51 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:36:51.886+0000 7f6e9a1cc700 -1 snap_mapper.add_oid found existing snaps mapped on 62:e97c6614:test-rados-api-smithi069-133790-78::foo:29, removing 2023-07-12T15:36:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:51 smithi145 ceph-mon[95801]: osdmap e426: 8 total, 8 up, 8 in 2023-07-12T15:36:53.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-77"}]: dispatch 2023-07-12T15:36:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:52 smithi069 ceph-mon[102777]: pgmap v620: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2023-07-12T15:36:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-77"}]: dispatch 2023-07-12T15:36:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:52 smithi069 ceph-mon[94241]: pgmap v620: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2023-07-12T15:36:53.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:36:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-77"}]: dispatch 2023-07-12T15:36:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:52 smithi145 ceph-mon[95801]: pgmap v620: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2023-07-12T15:36:54.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:53 smithi069 ceph-mon[102777]: osdmap e427: 8 total, 8 up, 8 in 2023-07-12T15:36:54.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:53 smithi069 ceph-mon[94241]: osdmap e427: 8 total, 8 up, 8 in 2023-07-12T15:36:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:53 smithi145 ceph-mon[95801]: osdmap e427: 8 total, 8 up, 8 in 2023-07-12T15:36:55.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:54 smithi069 ceph-mon[102777]: osdmap e428: 8 total, 8 up, 8 in 2023-07-12T15:36:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:54 smithi069 ceph-mon[102777]: pgmap v623: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-79","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:54 smithi069 ceph-mon[102777]: osdmap e429: 8 total, 8 up, 8 in 2023-07-12T15:36:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:54 smithi069 ceph-mon[94241]: osdmap e428: 8 total, 8 up, 8 in 2023-07-12T15:36:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:54 smithi069 ceph-mon[94241]: pgmap v623: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-79","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:54 smithi069 ceph-mon[94241]: osdmap e429: 8 total, 8 up, 8 in 2023-07-12T15:36:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:55.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:36:54 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:36:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:54 smithi145 ceph-mon[95801]: osdmap e428: 8 total, 8 up, 8 in 2023-07-12T15:36:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:36:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:54 smithi145 ceph-mon[95801]: pgmap v623: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:36:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-79","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:36:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:54 smithi145 ceph-mon[95801]: osdmap e429: 8 total, 8 up, 8 in 2023-07-12T15:36:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:36:56.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:55 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:56.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:55 smithi069 ceph-mon[94241]: osdmap e430: 8 total, 8 up, 8 in 2023-07-12T15:36:56.236 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:36:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:36:56] "GET /metrics HTTP/1.1" 200 228397 "" "Prometheus/2.43.0" 2023-07-12T15:36:56.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:55 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:56.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:55 smithi069 ceph-mon[102777]: osdmap e430: 8 total, 8 up, 8 in 2023-07-12T15:36:56.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:55 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:36:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:55 smithi145 ceph-mon[95801]: osdmap e430: 8 total, 8 up, 8 in 2023-07-12T15:36:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:56 smithi069 ceph-mon[102777]: pgmap v626: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:36:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:56 smithi069 ceph-mon[102777]: osdmap e431: 8 total, 8 up, 8 in 2023-07-12T15:36:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.d"}]: dispatch 2023-07-12T15:36:57.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:56 smithi069 ceph-mon[94241]: pgmap v626: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:36:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:56 smithi069 ceph-mon[94241]: osdmap e431: 8 total, 8 up, 8 in 2023-07-12T15:36:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.d"}]: dispatch 2023-07-12T15:36:57.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:56 smithi145 ceph-mon[95801]: pgmap v626: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:36:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:56 smithi145 ceph-mon[95801]: osdmap e431: 8 total, 8 up, 8 in 2023-07-12T15:36:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.d"}]: dispatch 2023-07-12T15:36:58.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:57 smithi069 ceph-mon[102777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.d"}]: dispatch 2023-07-12T15:36:58.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:57 smithi069 ceph-mon[102777]: 223.d deep-scrub starts 2023-07-12T15:36:58.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:57 smithi069 ceph-mon[102777]: 223.d deep-scrub ok 2023-07-12T15:36:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:57 smithi069 ceph-mon[94241]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.d"}]: dispatch 2023-07-12T15:36:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:57 smithi069 ceph-mon[94241]: 223.d deep-scrub starts 2023-07-12T15:36:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:57 smithi069 ceph-mon[94241]: 223.d deep-scrub ok 2023-07-12T15:36:58.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:57 smithi145 ceph-mon[95801]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.d"}]: dispatch 2023-07-12T15:36:58.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:57 smithi145 ceph-mon[95801]: 223.d deep-scrub starts 2023-07-12T15:36:58.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:57 smithi145 ceph-mon[95801]: 223.d deep-scrub ok 2023-07-12T15:36:59.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:58 smithi069 ceph-mon[94241]: pgmap v628: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 729 B/s wr, 2 op/s 2023-07-12T15:36:59.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:36:58 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:36:59.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:58 smithi069 ceph-mon[102777]: pgmap v628: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 729 B/s wr, 2 op/s 2023-07-12T15:36:59.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:36:58 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:36:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:58 smithi145 ceph-mon[95801]: pgmap v628: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 729 B/s wr, 2 op/s 2023-07-12T15:36:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:36:58 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:37:01.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:01 smithi069 ceph-mon[94241]: pgmap v629: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2023-07-12T15:37:01.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:01 smithi069 ceph-mon[102777]: pgmap v629: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2023-07-12T15:37:01.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:01 smithi145 ceph-mon[95801]: pgmap v629: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2023-07-12T15:37:03.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:02 smithi145 ceph-mon[95801]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 709 B/s rd, 709 B/s wr, 2 op/s 2023-07-12T15:37:03.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:02 smithi069 ceph-mon[94241]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 709 B/s rd, 709 B/s wr, 2 op/s 2023-07-12T15:37:03.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:02 smithi069 ceph-mon[102777]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 709 B/s rd, 709 B/s wr, 2 op/s 2023-07-12T15:37:05.167 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:37:04 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:37:05.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:05 smithi069 ceph-mon[94241]: pgmap v631: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 623 B/s rd, 623 B/s wr, 1 op/s 2023-07-12T15:37:05.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:05 smithi069 ceph-mon[102777]: pgmap v631: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 623 B/s rd, 623 B/s wr, 1 op/s 2023-07-12T15:37:05.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:05 smithi145 ceph-mon[95801]: pgmap v631: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 623 B/s rd, 623 B/s wr, 1 op/s 2023-07-12T15:37:06.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:06 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:06.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:06 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:06.487 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:37:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:37:06] "GET /metrics HTTP/1.1" 200 228411 "" "Prometheus/2.43.0" 2023-07-12T15:37:06.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:06 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:07.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:07 smithi069 ceph-mon[102777]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:37:07.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:07 smithi069 ceph-mon[94241]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:37:07.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:07 smithi145 ceph-mon[95801]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:37:09.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:09 smithi069 ceph-mon[102777]: pgmap v633: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 913 B/s rd, 456 B/s wr, 1 op/s 2023-07-12T15:37:09.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:09 smithi069 ceph-mon[94241]: pgmap v633: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 913 B/s rd, 456 B/s wr, 1 op/s 2023-07-12T15:37:09.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:09 smithi145 ceph-mon[95801]: pgmap v633: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 913 B/s rd, 456 B/s wr, 1 op/s 2023-07-12T15:37:11.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:11 smithi069 ceph-mon[102777]: pgmap v634: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2023-07-12T15:37:11.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:11 smithi069 ceph-mon[94241]: pgmap v634: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2023-07-12T15:37:11.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:11 smithi145 ceph-mon[95801]: pgmap v634: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2023-07-12T15:37:13.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:12 smithi145 ceph-mon[95801]: pgmap v635: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:37:13.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:12 smithi069 ceph-mon[102777]: pgmap v635: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:37:13.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:12 smithi069 ceph-mon[94241]: pgmap v635: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:37:14.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:37:14.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:37:14.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:37:15.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:14 smithi069 ceph-mon[102777]: pgmap v636: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:37:15.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:14 smithi069 ceph-mon[94241]: pgmap v636: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:37:15.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:37:14 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:37:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:14 smithi145 ceph-mon[95801]: pgmap v636: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:37:16.148 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:15 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:16.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:15 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:16.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:15 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:16.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:37:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:37:16] "GET /metrics HTTP/1.1" 200 228371 "" "Prometheus/2.43.0" 2023-07-12T15:37:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:16 smithi069 ceph-mon[102777]: pgmap v637: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:37:17.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:16 smithi069 ceph-mon[94241]: pgmap v637: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:37:17.237 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:37:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-0[107243]: 2023-07-12T15:37:16.944+0000 7f5e9d3e3700 -1 snap_mapper.add_oid found existing snaps mapped on 223:b66b7027:test-rados-api-smithi069-133790-80::foo:2, removing 2023-07-12T15:37:17.237 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:37:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-1[111786]: 2023-07-12T15:37:16.944+0000 7fab389ed700 -1 snap_mapper.add_oid found existing snaps mapped on 223:b66b7027:test-rados-api-smithi069-133790-80::foo:2, removing 2023-07-12T15:37:17.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:16 smithi145 ceph-mon[95801]: pgmap v637: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:37:17.351 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:37:16 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:37:16.943+0000 7f55c1685700 -1 snap_mapper.add_oid found existing snaps mapped on 223:b66b7027:test-rados-api-smithi069-133790-80::foo:2, removing 2023-07-12T15:37:18.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:18.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-79"}]: dispatch 2023-07-12T15:37:18.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:18.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-79"}]: dispatch 2023-07-12T15:37:18.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-79"}]: dispatch 2023-07-12T15:37:19.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:18 smithi069 ceph-mon[102777]: osdmap e432: 8 total, 8 up, 8 in 2023-07-12T15:37:19.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:18 smithi069 ceph-mon[102777]: pgmap v639: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:37:19.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:18 smithi069 ceph-mon[94241]: osdmap e432: 8 total, 8 up, 8 in 2023-07-12T15:37:19.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:18 smithi069 ceph-mon[94241]: pgmap v639: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:37:19.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:18 smithi145 ceph-mon[95801]: osdmap e432: 8 total, 8 up, 8 in 2023-07-12T15:37:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:18 smithi145 ceph-mon[95801]: pgmap v639: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:37:20.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:19 smithi069 ceph-mon[102777]: osdmap e433: 8 total, 8 up, 8 in 2023-07-12T15:37:20.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:20.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:19 smithi069 ceph-mon[94241]: osdmap e433: 8 total, 8 up, 8 in 2023-07-12T15:37:20.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:20.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:19 smithi145 ceph-mon[95801]: osdmap e433: 8 total, 8 up, 8 in 2023-07-12T15:37:20.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:21.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-81","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:21.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[102777]: osdmap e434: 8 total, 8 up, 8 in 2023-07-12T15:37:21.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:21.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:21.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[102777]: pgmap v642: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2023-07-12T15:37:21.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:21.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[102777]: osdmap e435: 8 total, 8 up, 8 in 2023-07-12T15:37:21.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:21.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-81","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:21.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[94241]: osdmap e434: 8 total, 8 up, 8 in 2023-07-12T15:37:21.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:21.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:21.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[94241]: pgmap v642: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2023-07-12T15:37:21.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:21.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[94241]: osdmap e435: 8 total, 8 up, 8 in 2023-07-12T15:37:21.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:21.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-81","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:20 smithi145 ceph-mon[95801]: osdmap e434: 8 total, 8 up, 8 in 2023-07-12T15:37:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:21.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:21.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:20 smithi145 ceph-mon[95801]: pgmap v642: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2023-07-12T15:37:21.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:21.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:20 smithi145 ceph-mon[95801]: osdmap e435: 8 total, 8 up, 8 in 2023-07-12T15:37:21.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:23.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:37:23.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:22 smithi069 ceph-mon[102777]: osdmap e436: 8 total, 8 up, 8 in 2023-07-12T15:37:23.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:37:23.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:22 smithi069 ceph-mon[102777]: pgmap v645: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2023-07-12T15:37:23.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:37:23.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:22 smithi069 ceph-mon[94241]: osdmap e436: 8 total, 8 up, 8 in 2023-07-12T15:37:23.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:37:23.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:22 smithi069 ceph-mon[94241]: pgmap v645: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2023-07-12T15:37:23.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:37:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:22 smithi145 ceph-mon[95801]: osdmap e436: 8 total, 8 up, 8 in 2023-07-12T15:37:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:37:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:22 smithi145 ceph-mon[95801]: pgmap v645: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2023-07-12T15:37:24.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:37:24.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[102777]: osdmap e437: 8 total, 8 up, 8 in 2023-07-12T15:37:24.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:24.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:37:24.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:37:24.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:37:24.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:37:24.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:37:24.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[94241]: osdmap e437: 8 total, 8 up, 8 in 2023-07-12T15:37:24.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:24.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:37:24.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:37:24.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:37:24.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:23 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:37:24.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:37:24.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:23 smithi145 ceph-mon[95801]: osdmap e437: 8 total, 8 up, 8 in 2023-07-12T15:37:24.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:24.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:23 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:37:24.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:23 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:37:24.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:23 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:37:24.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:23 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:37:25.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:25.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:24 smithi069 ceph-mon[102777]: osdmap e438: 8 total, 8 up, 8 in 2023-07-12T15:37:25.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2023-07-12T15:37:25.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:24 smithi069 ceph-mon[102777]: pgmap v648: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:37:25.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:25.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:24 smithi069 ceph-mon[94241]: osdmap e438: 8 total, 8 up, 8 in 2023-07-12T15:37:25.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2023-07-12T15:37:25.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:24 smithi069 ceph-mon[94241]: pgmap v648: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:37:25.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:37:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:37:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:24 smithi145 ceph-mon[95801]: osdmap e438: 8 total, 8 up, 8 in 2023-07-12T15:37:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2023-07-12T15:37:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:24 smithi145 ceph-mon[95801]: pgmap v648: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:37:26.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:37:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:37:26] "GET /metrics HTTP/1.1" 200 228371 "" "Prometheus/2.43.0" 2023-07-12T15:37:26.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:25 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:26.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2023-07-12T15:37:26.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:25 smithi069 ceph-mon[94241]: osdmap e439: 8 total, 8 up, 8 in 2023-07-12T15:37:26.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2023-07-12T15:37:26.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:25 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:26.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2023-07-12T15:37:26.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:25 smithi069 ceph-mon[102777]: osdmap e439: 8 total, 8 up, 8 in 2023-07-12T15:37:26.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2023-07-12T15:37:26.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:25 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:26.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2023-07-12T15:37:26.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:25 smithi145 ceph-mon[95801]: osdmap e439: 8 total, 8 up, 8 in 2023-07-12T15:37:26.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2023-07-12T15:37:27.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2023-07-12T15:37:27.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:26 smithi069 ceph-mon[102777]: osdmap e440: 8 total, 8 up, 8 in 2023-07-12T15:37:27.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:27.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:26 smithi069 ceph-mon[102777]: pgmap v651: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2023-07-12T15:37:27.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2023-07-12T15:37:27.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:26 smithi069 ceph-mon[94241]: osdmap e440: 8 total, 8 up, 8 in 2023-07-12T15:37:27.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:27.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:26 smithi069 ceph-mon[94241]: pgmap v651: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2023-07-12T15:37:27.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2023-07-12T15:37:27.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:26 smithi145 ceph-mon[95801]: osdmap e440: 8 total, 8 up, 8 in 2023-07-12T15:37:27.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:27.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:26 smithi145 ceph-mon[95801]: pgmap v651: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2023-07-12T15:37:28.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:28.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:27 smithi069 ceph-mon[102777]: osdmap e441: 8 total, 8 up, 8 in 2023-07-12T15:37:28.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:28.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-81"}]: dispatch 2023-07-12T15:37:28.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:28.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:27 smithi069 ceph-mon[94241]: osdmap e441: 8 total, 8 up, 8 in 2023-07-12T15:37:28.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:28.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-81"}]: dispatch 2023-07-12T15:37:28.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:28.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:27 smithi145 ceph-mon[95801]: osdmap e441: 8 total, 8 up, 8 in 2023-07-12T15:37:28.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:28.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-81"}]: dispatch 2023-07-12T15:37:29.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:28 smithi069 ceph-mon[102777]: osdmap e442: 8 total, 8 up, 8 in 2023-07-12T15:37:29.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:28 smithi069 ceph-mon[102777]: pgmap v654: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2023-07-12T15:37:29.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:28 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:37:29.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:28 smithi069 ceph-mon[94241]: osdmap e442: 8 total, 8 up, 8 in 2023-07-12T15:37:29.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:28 smithi069 ceph-mon[94241]: pgmap v654: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2023-07-12T15:37:29.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:28 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:37:29.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:28 smithi145 ceph-mon[95801]: osdmap e442: 8 total, 8 up, 8 in 2023-07-12T15:37:29.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:28 smithi145 ceph-mon[95801]: pgmap v654: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2023-07-12T15:37:29.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:28 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:37:30.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:29 smithi069 ceph-mon[102777]: osdmap e443: 8 total, 8 up, 8 in 2023-07-12T15:37:30.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:30.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:29 smithi069 ceph-mon[94241]: osdmap e443: 8 total, 8 up, 8 in 2023-07-12T15:37:30.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:30.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:29 smithi145 ceph-mon[95801]: osdmap e443: 8 total, 8 up, 8 in 2023-07-12T15:37:30.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:31.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-83","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:31.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:30 smithi069 ceph-mon[94241]: osdmap e444: 8 total, 8 up, 8 in 2023-07-12T15:37:31.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:31.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:31.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:30 smithi069 ceph-mon[94241]: pgmap v657: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2023-07-12T15:37:31.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-83","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:31.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:30 smithi069 ceph-mon[102777]: osdmap e444: 8 total, 8 up, 8 in 2023-07-12T15:37:31.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:31.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:31.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:30 smithi069 ceph-mon[102777]: pgmap v657: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2023-07-12T15:37:31.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-83","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:31.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:30 smithi145 ceph-mon[95801]: osdmap e444: 8 total, 8 up, 8 in 2023-07-12T15:37:31.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:31.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:31.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:30 smithi145 ceph-mon[95801]: pgmap v657: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2023-07-12T15:37:32.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:32.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:31 smithi069 ceph-mon[94241]: osdmap e445: 8 total, 8 up, 8 in 2023-07-12T15:37:32.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:32.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:32.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:31 smithi069 ceph-mon[102777]: osdmap e445: 8 total, 8 up, 8 in 2023-07-12T15:37:32.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:32.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:32.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:31 smithi145 ceph-mon[95801]: osdmap e445: 8 total, 8 up, 8 in 2023-07-12T15:37:32.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:33.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:37:33.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:32 smithi145 ceph-mon[95801]: osdmap e446: 8 total, 8 up, 8 in 2023-07-12T15:37:33.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:37:33.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:32 smithi145 ceph-mon[95801]: pgmap v660: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2023-07-12T15:37:33.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:37:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:32 smithi069 ceph-mon[102777]: osdmap e446: 8 total, 8 up, 8 in 2023-07-12T15:37:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:37:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:32 smithi069 ceph-mon[102777]: pgmap v660: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2023-07-12T15:37:33.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:37:33.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:32 smithi069 ceph-mon[94241]: osdmap e446: 8 total, 8 up, 8 in 2023-07-12T15:37:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:37:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:32 smithi069 ceph-mon[94241]: pgmap v660: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2023-07-12T15:37:34.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:37:34.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:33 smithi145 ceph-mon[95801]: osdmap e447: 8 total, 8 up, 8 in 2023-07-12T15:37:34.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:34.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:37:34.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:33 smithi069 ceph-mon[102777]: osdmap e447: 8 total, 8 up, 8 in 2023-07-12T15:37:34.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:34.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:37:34.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:33 smithi069 ceph-mon[94241]: osdmap e447: 8 total, 8 up, 8 in 2023-07-12T15:37:34.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:35.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:37:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:37:35.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:35.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:34 smithi145 ceph-mon[95801]: osdmap e448: 8 total, 8 up, 8 in 2023-07-12T15:37:35.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:34 smithi145 ceph-mon[95801]: pgmap v663: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:37:35.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:35.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:34 smithi069 ceph-mon[102777]: osdmap e448: 8 total, 8 up, 8 in 2023-07-12T15:37:35.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:34 smithi069 ceph-mon[102777]: pgmap v663: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:37:35.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:35.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:34 smithi069 ceph-mon[94241]: osdmap e448: 8 total, 8 up, 8 in 2023-07-12T15:37:35.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:34 smithi069 ceph-mon[94241]: pgmap v663: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:37:36.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:35 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:36.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:35 smithi145 ceph-mon[95801]: osdmap e449: 8 total, 8 up, 8 in 2023-07-12T15:37:36.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:37:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:37:36] "GET /metrics HTTP/1.1" 200 225956 "" "Prometheus/2.43.0" 2023-07-12T15:37:36.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:35 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:36.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:35 smithi069 ceph-mon[102777]: osdmap e449: 8 total, 8 up, 8 in 2023-07-12T15:37:36.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:35 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:36.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:35 smithi069 ceph-mon[94241]: osdmap e449: 8 total, 8 up, 8 in 2023-07-12T15:37:37.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:37 smithi145 ceph-mon[95801]: osdmap e450: 8 total, 8 up, 8 in 2023-07-12T15:37:37.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:37.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-83"}]: dispatch 2023-07-12T15:37:37.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:37 smithi145 ceph-mon[95801]: pgmap v666: 228 pgs: 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2023-07-12T15:37:37.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:37 smithi069 ceph-mon[94241]: osdmap e450: 8 total, 8 up, 8 in 2023-07-12T15:37:37.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:37.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-83"}]: dispatch 2023-07-12T15:37:37.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:37 smithi069 ceph-mon[94241]: pgmap v666: 228 pgs: 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2023-07-12T15:37:37.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:37 smithi069 ceph-mon[102777]: osdmap e450: 8 total, 8 up, 8 in 2023-07-12T15:37:37.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:37.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-83"}]: dispatch 2023-07-12T15:37:37.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:37 smithi069 ceph-mon[102777]: pgmap v666: 228 pgs: 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2023-07-12T15:37:38.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:38 smithi145 ceph-mon[95801]: osdmap e451: 8 total, 8 up, 8 in 2023-07-12T15:37:38.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:38 smithi069 ceph-mon[102777]: osdmap e451: 8 total, 8 up, 8 in 2023-07-12T15:37:38.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:38 smithi069 ceph-mon[94241]: osdmap e451: 8 total, 8 up, 8 in 2023-07-12T15:37:39.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:39 smithi145 ceph-mon[95801]: osdmap e452: 8 total, 8 up, 8 in 2023-07-12T15:37:39.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:39.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:39 smithi145 ceph-mon[95801]: pgmap v669: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:37:39.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-85","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:39.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:39 smithi145 ceph-mon[95801]: osdmap e453: 8 total, 8 up, 8 in 2023-07-12T15:37:39.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:39 smithi069 ceph-mon[102777]: osdmap e452: 8 total, 8 up, 8 in 2023-07-12T15:37:39.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:39.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:39 smithi069 ceph-mon[102777]: pgmap v669: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:37:39.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-85","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:39.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:39 smithi069 ceph-mon[102777]: osdmap e453: 8 total, 8 up, 8 in 2023-07-12T15:37:39.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:39 smithi069 ceph-mon[94241]: osdmap e452: 8 total, 8 up, 8 in 2023-07-12T15:37:39.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:39.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:39 smithi069 ceph-mon[94241]: pgmap v669: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:37:39.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-85","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:39.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:39 smithi069 ceph-mon[94241]: osdmap e453: 8 total, 8 up, 8 in 2023-07-12T15:37:40.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:40 smithi145 ceph-mon[95801]: osdmap e454: 8 total, 8 up, 8 in 2023-07-12T15:37:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:40 smithi069 ceph-mon[102777]: osdmap e454: 8 total, 8 up, 8 in 2023-07-12T15:37:40.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:40 smithi069 ceph-mon[94241]: osdmap e454: 8 total, 8 up, 8 in 2023-07-12T15:37:40.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:41.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:41 smithi145 ceph-mon[95801]: pgmap v672: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2023-07-12T15:37:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:37:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:41 smithi145 ceph-mon[95801]: osdmap e455: 8 total, 8 up, 8 in 2023-07-12T15:37:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:37:41.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:41 smithi069 ceph-mon[102777]: pgmap v672: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2023-07-12T15:37:41.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:37:41.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:41 smithi069 ceph-mon[102777]: osdmap e455: 8 total, 8 up, 8 in 2023-07-12T15:37:41.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:37:41.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:41 smithi069 ceph-mon[94241]: pgmap v672: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2023-07-12T15:37:41.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_tier","val": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:37:41.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:41 smithi069 ceph-mon[94241]: osdmap e455: 8 total, 8 up, 8 in 2023-07-12T15:37:41.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:37:43.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:37:43.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:43 smithi145 ceph-mon[95801]: osdmap e456: 8 total, 8 up, 8 in 2023-07-12T15:37:43.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:43.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:43 smithi145 ceph-mon[95801]: pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2023-07-12T15:37:43.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:37:43.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:43 smithi069 ceph-mon[102777]: osdmap e456: 8 total, 8 up, 8 in 2023-07-12T15:37:43.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:43.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:43 smithi069 ceph-mon[102777]: pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2023-07-12T15:37:43.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:37:43.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:43 smithi069 ceph-mon[94241]: osdmap e456: 8 total, 8 up, 8 in 2023-07-12T15:37:43.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:43.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:43 smithi069 ceph-mon[94241]: pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2023-07-12T15:37:44.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:44.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:44 smithi145 ceph-mon[95801]: osdmap e457: 8 total, 8 up, 8 in 2023-07-12T15:37:44.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:44 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:37:44.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:44.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:44 smithi069 ceph-mon[102777]: osdmap e457: 8 total, 8 up, 8 in 2023-07-12T15:37:44.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:44 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:37:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:44 smithi069 ceph-mon[94241]: osdmap e457: 8 total, 8 up, 8 in 2023-07-12T15:37:44.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:44 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:37:45.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:37:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:37:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:45 smithi145 ceph-mon[95801]: osdmap e458: 8 total, 8 up, 8 in 2023-07-12T15:37:45.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:45 smithi145 ceph-mon[95801]: pgmap v678: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:37:45.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:45 smithi069 ceph-mon[102777]: osdmap e458: 8 total, 8 up, 8 in 2023-07-12T15:37:45.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:45 smithi069 ceph-mon[102777]: pgmap v678: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:37:45.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:45 smithi069 ceph-mon[94241]: osdmap e458: 8 total, 8 up, 8 in 2023-07-12T15:37:45.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:45 smithi069 ceph-mon[94241]: pgmap v678: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:37:46.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:46 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:46 smithi145 ceph-mon[95801]: osdmap e459: 8 total, 8 up, 8 in 2023-07-12T15:37:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-85"}]: dispatch 2023-07-12T15:37:46.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:37:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:37:46] "GET /metrics HTTP/1.1" 200 228426 "" "Prometheus/2.43.0" 2023-07-12T15:37:46.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:46 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:46.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:46 smithi069 ceph-mon[102777]: osdmap e459: 8 total, 8 up, 8 in 2023-07-12T15:37:46.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:46.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-85"}]: dispatch 2023-07-12T15:37:46.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:46 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:46.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:46 smithi069 ceph-mon[94241]: osdmap e459: 8 total, 8 up, 8 in 2023-07-12T15:37:46.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:46.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-85"}]: dispatch 2023-07-12T15:37:47.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:47 smithi145 ceph-mon[95801]: osdmap e460: 8 total, 8 up, 8 in 2023-07-12T15:37:47.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:47 smithi145 ceph-mon[95801]: pgmap v681: 196 pgs: 196 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:37:47.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:47 smithi069 ceph-mon[102777]: osdmap e460: 8 total, 8 up, 8 in 2023-07-12T15:37:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:47 smithi069 ceph-mon[102777]: pgmap v681: 196 pgs: 196 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:37:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:47 smithi069 ceph-mon[94241]: osdmap e460: 8 total, 8 up, 8 in 2023-07-12T15:37:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:47 smithi069 ceph-mon[94241]: pgmap v681: 196 pgs: 196 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:37:48.068 INFO:tasks.workunit.client.0.smithi069.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (17820 ms) 2023-07-12T15:37:48.068 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-07-12T15:37:48.068 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22669 ms) 2023-07-12T15:37:48.068 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-07-12T15:37:48.068 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3011 ms) 2023-07-12T15:37:48.069 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-07-12T15:37:48.069 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.069 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3020 ms) 2023-07-12T15:37:48.069 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-07-12T15:37:48.069 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3010 ms) 2023-07-12T15:37:48.069 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-07-12T15:37:48.069 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.070 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3004 ms) 2023-07-12T15:37:48.070 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-07-12T15:37:48.070 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.070 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4030 ms) 2023-07-12T15:37:48.070 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-07-12T15:37:48.070 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.070 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38249 ms) 2023-07-12T15:37:48.071 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-07-12T15:37:48.071 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.071 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16990 ms) 2023-07-12T15:37:48.071 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-07-12T15:37:48.071 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.071 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4014 ms) 2023-07-12T15:37:48.071 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-07-12T15:37:48.072 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.072 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3013 ms) 2023-07-12T15:37:48.072 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-07-12T15:37:48.073 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.073 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24982 ms) 2023-07-12T15:37:48.073 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-07-12T15:37:48.073 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.073 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (14689 ms) 2023-07-12T15:37:48.073 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-07-12T15:37:48.073 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5015 ms) 2023-07-12T15:37:48.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-07-12T15:37:48.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4031 ms) 2023-07-12T15:37:48.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-07-12T15:37:48.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.074 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrubs... 2023-07-12T15:37:48.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2023-07-12T15:37:48.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (24981 ms) 2023-07-12T15:37:48.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-07-12T15:37:48.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10072 ms) 2023-07-12T15:37:48.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-07-12T15:37:48.075 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9063 ms) 2023-07-12T15:37:48.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-07-12T15:37:48.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:37:48.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9039 ms) 2023-07-12T15:37:48.076 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-07-12T15:37:48.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:48 smithi145 ceph-mon[95801]: osdmap e461: 8 total, 8 up, 8 in 2023-07-12T15:37:48.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:48.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:48 smithi069 ceph-mon[102777]: osdmap e461: 8 total, 8 up, 8 in 2023-07-12T15:37:48.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:48.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:48 smithi069 ceph-mon[94241]: osdmap e461: 8 total, 8 up, 8 in 2023-07-12T15:37:48.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:49.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-87","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:49 smithi145 ceph-mon[95801]: osdmap e462: 8 total, 8 up, 8 in 2023-07-12T15:37:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:49 smithi145 ceph-mon[95801]: pgmap v684: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:37:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:49 smithi145 ceph-mon[95801]: osdmap e463: 8 total, 8 up, 8 in 2023-07-12T15:37:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:49.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-87","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:49.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:49 smithi069 ceph-mon[102777]: osdmap e462: 8 total, 8 up, 8 in 2023-07-12T15:37:49.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:49.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:49 smithi069 ceph-mon[102777]: pgmap v684: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:37:49.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:49 smithi069 ceph-mon[102777]: osdmap e463: 8 total, 8 up, 8 in 2023-07-12T15:37:49.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-87","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:49 smithi069 ceph-mon[94241]: osdmap e462: 8 total, 8 up, 8 in 2023-07-12T15:37:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:49 smithi069 ceph-mon[94241]: pgmap v684: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:37:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:49 smithi069 ceph-mon[94241]: osdmap e463: 8 total, 8 up, 8 in 2023-07-12T15:37:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:51 smithi145 ceph-mon[95801]: osdmap e464: 8 total, 8 up, 8 in 2023-07-12T15:37:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_tier","val": "test-rados-api-smithi069-133790-89-test-flush"}]: dispatch 2023-07-12T15:37:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:51 smithi145 ceph-mon[95801]: pgmap v687: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 4.5 KiB/s wr, 189 op/s 2023-07-12T15:37:51.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:51.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:51 smithi069 ceph-mon[102777]: osdmap e464: 8 total, 8 up, 8 in 2023-07-12T15:37:51.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_tier","val": "test-rados-api-smithi069-133790-89-test-flush"}]: dispatch 2023-07-12T15:37:51.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:51 smithi069 ceph-mon[102777]: pgmap v687: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 4.5 KiB/s wr, 189 op/s 2023-07-12T15:37:51.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:51.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:51 smithi069 ceph-mon[94241]: osdmap e464: 8 total, 8 up, 8 in 2023-07-12T15:37:51.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_tier","val": "test-rados-api-smithi069-133790-89-test-flush"}]: dispatch 2023-07-12T15:37:51.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:51 smithi069 ceph-mon[94241]: pgmap v687: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 4.5 KiB/s wr, 189 op/s 2023-07-12T15:37:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_tier","val": "test-rados-api-smithi069-133790-89-test-flush"}]': finished 2023-07-12T15:37:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:52 smithi145 ceph-mon[95801]: osdmap e465: 8 total, 8 up, 8 in 2023-07-12T15:37:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:37:52.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_tier","val": "test-rados-api-smithi069-133790-89-test-flush"}]': finished 2023-07-12T15:37:52.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:52 smithi069 ceph-mon[102777]: osdmap e465: 8 total, 8 up, 8 in 2023-07-12T15:37:52.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:37:52.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_tier","val": "test-rados-api-smithi069-133790-89-test-flush"}]': finished 2023-07-12T15:37:52.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:52 smithi069 ceph-mon[94241]: osdmap e465: 8 total, 8 up, 8 in 2023-07-12T15:37:52.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:37:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:37:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:53 smithi145 ceph-mon[95801]: osdmap e466: 8 total, 8 up, 8 in 2023-07-12T15:37:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:53 smithi145 ceph-mon[95801]: pgmap v690: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 4.5 KiB/s wr, 189 op/s 2023-07-12T15:37:53.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:37:53.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:53 smithi069 ceph-mon[102777]: osdmap e466: 8 total, 8 up, 8 in 2023-07-12T15:37:53.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:53.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:53 smithi069 ceph-mon[102777]: pgmap v690: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 4.5 KiB/s wr, 189 op/s 2023-07-12T15:37:53.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:37:53.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:53 smithi069 ceph-mon[94241]: osdmap e466: 8 total, 8 up, 8 in 2023-07-12T15:37:53.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:37:53.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:53 smithi069 ceph-mon[94241]: pgmap v690: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 4.5 KiB/s wr, 189 op/s 2023-07-12T15:37:54.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:54 smithi145 ceph-mon[95801]: osdmap e467: 8 total, 8 up, 8 in 2023-07-12T15:37:54.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:54.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:54 smithi069 ceph-mon[94241]: osdmap e467: 8 total, 8 up, 8 in 2023-07-12T15:37:54.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:37:54.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:54 smithi069 ceph-mon[102777]: osdmap e467: 8 total, 8 up, 8 in 2023-07-12T15:37:55.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:37:54 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:37:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:55 smithi145 ceph-mon[95801]: osdmap e468: 8 total, 8 up, 8 in 2023-07-12T15:37:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:55 smithi145 ceph-mon[95801]: pgmap v693: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:37:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-87"}]: dispatch 2023-07-12T15:37:55.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:55 smithi069 ceph-mon[102777]: osdmap e468: 8 total, 8 up, 8 in 2023-07-12T15:37:55.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:55 smithi069 ceph-mon[102777]: pgmap v693: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:37:55.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:55.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-87"}]: dispatch 2023-07-12T15:37:55.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:55 smithi069 ceph-mon[94241]: osdmap e468: 8 total, 8 up, 8 in 2023-07-12T15:37:55.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:55 smithi069 ceph-mon[94241]: pgmap v693: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:37:55.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:37:55.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-87"}]: dispatch 2023-07-12T15:37:56.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:37:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:37:56] "GET /metrics HTTP/1.1" 200 228426 "" "Prometheus/2.43.0" 2023-07-12T15:37:56.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:56 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:56.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:56 smithi069 ceph-mon[94241]: osdmap e469: 8 total, 8 up, 8 in 2023-07-12T15:37:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:56 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:56.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:56 smithi069 ceph-mon[102777]: osdmap e469: 8 total, 8 up, 8 in 2023-07-12T15:37:56.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:56 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:37:56.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:56 smithi145 ceph-mon[95801]: osdmap e469: 8 total, 8 up, 8 in 2023-07-12T15:37:57.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:57 smithi069 ceph-mon[102777]: osdmap e470: 8 total, 8 up, 8 in 2023-07-12T15:37:57.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:57 smithi069 ceph-mon[102777]: pgmap v696: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:37:57.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:57.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-90","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:57.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:57 smithi069 ceph-mon[102777]: osdmap e471: 8 total, 8 up, 8 in 2023-07-12T15:37:57.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:57.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:57 smithi069 ceph-mon[94241]: osdmap e470: 8 total, 8 up, 8 in 2023-07-12T15:37:57.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:57 smithi069 ceph-mon[94241]: pgmap v696: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:37:57.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:57.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-90","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:57.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:57 smithi069 ceph-mon[94241]: osdmap e471: 8 total, 8 up, 8 in 2023-07-12T15:37:57.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:57.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:57 smithi145 ceph-mon[95801]: osdmap e470: 8 total, 8 up, 8 in 2023-07-12T15:37:57.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:57 smithi145 ceph-mon[95801]: pgmap v696: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:37:57.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:37:57.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-90","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:37:57.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:57 smithi145 ceph-mon[95801]: osdmap e471: 8 total, 8 up, 8 in 2023-07-12T15:37:57.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:37:58.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:58.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:58.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:58 smithi069 ceph-mon[94241]: osdmap e472: 8 total, 8 up, 8 in 2023-07-12T15:37:58.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:58.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:58.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:58 smithi069 ceph-mon[102777]: osdmap e472: 8 total, 8 up, 8 in 2023-07-12T15:37:58.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:37:58.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:37:58.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:58 smithi145 ceph-mon[95801]: osdmap e472: 8 total, 8 up, 8 in 2023-07-12T15:37:59.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:59 smithi069 ceph-mon[102777]: pgmap v698: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 254 B/s wr, 2 op/s 2023-07-12T15:37:59.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:37:59 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:37:59.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:59 smithi069 ceph-mon[94241]: pgmap v698: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 254 B/s wr, 2 op/s 2023-07-12T15:37:59.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:37:59 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:37:59.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:59 smithi145 ceph-mon[95801]: pgmap v698: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 254 B/s wr, 2 op/s 2023-07-12T15:37:59.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:37:59 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:38:00.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:00 smithi069 ceph-mon[102777]: osdmap e473: 8 total, 8 up, 8 in 2023-07-12T15:38:00.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:00 smithi069 ceph-mon[94241]: osdmap e473: 8 total, 8 up, 8 in 2023-07-12T15:38:00.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:00 smithi145 ceph-mon[95801]: osdmap e473: 8 total, 8 up, 8 in 2023-07-12T15:38:01.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:01 smithi069 ceph-mon[94241]: pgmap v701: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2023-07-12T15:38:01.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:01 smithi069 ceph-mon[94241]: osdmap e474: 8 total, 8 up, 8 in 2023-07-12T15:38:01.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:01.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-90"}]: dispatch 2023-07-12T15:38:01.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:01 smithi069 ceph-mon[102777]: pgmap v701: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2023-07-12T15:38:01.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:01 smithi069 ceph-mon[102777]: osdmap e474: 8 total, 8 up, 8 in 2023-07-12T15:38:01.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:01.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-90"}]: dispatch 2023-07-12T15:38:01.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:01 smithi145 ceph-mon[95801]: pgmap v701: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2023-07-12T15:38:01.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:01 smithi145 ceph-mon[95801]: osdmap e474: 8 total, 8 up, 8 in 2023-07-12T15:38:01.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:01.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-90"}]: dispatch 2023-07-12T15:38:02.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:02 smithi069 ceph-mon[94241]: osdmap e475: 8 total, 8 up, 8 in 2023-07-12T15:38:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:02 smithi069 ceph-mon[102777]: osdmap e475: 8 total, 8 up, 8 in 2023-07-12T15:38:02.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:02 smithi145 ceph-mon[95801]: osdmap e475: 8 total, 8 up, 8 in 2023-07-12T15:38:03.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:03 smithi069 ceph-mon[102777]: pgmap v704: 196 pgs: 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:38:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:03 smithi069 ceph-mon[102777]: osdmap e476: 8 total, 8 up, 8 in 2023-07-12T15:38:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:38:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-92","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:38:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:03 smithi069 ceph-mon[94241]: pgmap v704: 196 pgs: 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:38:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:03 smithi069 ceph-mon[94241]: osdmap e476: 8 total, 8 up, 8 in 2023-07-12T15:38:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:38:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-92","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:38:03.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:03 smithi145 ceph-mon[95801]: pgmap v704: 196 pgs: 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:38:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:03 smithi145 ceph-mon[95801]: osdmap e476: 8 total, 8 up, 8 in 2023-07-12T15:38:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:38:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-92","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:38:04.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:04 smithi069 ceph-mon[102777]: osdmap e477: 8 total, 8 up, 8 in 2023-07-12T15:38:04.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:38:04.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:04 smithi069 ceph-mon[94241]: osdmap e477: 8 total, 8 up, 8 in 2023-07-12T15:38:04.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:38:04.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:04 smithi145 ceph-mon[95801]: osdmap e477: 8 total, 8 up, 8 in 2023-07-12T15:38:04.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:38:05.212 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:38:04 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:38:05.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:05 smithi069 ceph-mon[94241]: pgmap v707: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:38:05.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:05 smithi069 ceph-mon[94241]: osdmap e478: 8 total, 8 up, 8 in 2023-07-12T15:38:05.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:05 smithi069 ceph-mon[102777]: pgmap v707: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:38:05.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:05 smithi069 ceph-mon[102777]: osdmap e478: 8 total, 8 up, 8 in 2023-07-12T15:38:05.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:05 smithi145 ceph-mon[95801]: pgmap v707: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:38:05.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:05 smithi145 ceph-mon[95801]: osdmap e478: 8 total, 8 up, 8 in 2023-07-12T15:38:06.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:38:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:38:06] "GET /metrics HTTP/1.1" 200 228441 "" "Prometheus/2.43.0" 2023-07-12T15:38:06.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:06 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:06.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:06 smithi069 ceph-mon[102777]: osdmap e479: 8 total, 8 up, 8 in 2023-07-12T15:38:06.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:06.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-92"}]: dispatch 2023-07-12T15:38:06.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:06 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:06.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:06 smithi069 ceph-mon[94241]: osdmap e479: 8 total, 8 up, 8 in 2023-07-12T15:38:06.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:06.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-92"}]: dispatch 2023-07-12T15:38:06.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:06 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:06.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:06 smithi145 ceph-mon[95801]: osdmap e479: 8 total, 8 up, 8 in 2023-07-12T15:38:06.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:06.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-92"}]: dispatch 2023-07-12T15:38:07.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:07 smithi069 ceph-mon[102777]: pgmap v710: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2023-07-12T15:38:07.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:07 smithi069 ceph-mon[102777]: osdmap e480: 8 total, 8 up, 8 in 2023-07-12T15:38:07.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:07 smithi069 ceph-mon[94241]: pgmap v710: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2023-07-12T15:38:07.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:07 smithi069 ceph-mon[94241]: osdmap e480: 8 total, 8 up, 8 in 2023-07-12T15:38:07.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:07 smithi145 ceph-mon[95801]: pgmap v710: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2023-07-12T15:38:07.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:07 smithi145 ceph-mon[95801]: osdmap e480: 8 total, 8 up, 8 in 2023-07-12T15:38:08.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:08 smithi069 ceph-mon[102777]: osdmap e481: 8 total, 8 up, 8 in 2023-07-12T15:38:08.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:38:08.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:08 smithi069 ceph-mon[94241]: osdmap e481: 8 total, 8 up, 8 in 2023-07-12T15:38:08.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:38:08.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:08 smithi145 ceph-mon[95801]: osdmap e481: 8 total, 8 up, 8 in 2023-07-12T15:38:08.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:38:09.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:09 smithi145 ceph-mon[95801]: pgmap v713: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:38:09.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-94","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:38:09.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:09 smithi145 ceph-mon[95801]: osdmap e482: 8 total, 8 up, 8 in 2023-07-12T15:38:09.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:38:09.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:09 smithi069 ceph-mon[102777]: pgmap v713: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:38:09.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-94","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:38:09.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:09 smithi069 ceph-mon[102777]: osdmap e482: 8 total, 8 up, 8 in 2023-07-12T15:38:09.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:38:09.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:09 smithi069 ceph-mon[94241]: pgmap v713: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:38:09.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-94","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:38:09.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:09 smithi069 ceph-mon[94241]: osdmap e482: 8 total, 8 up, 8 in 2023-07-12T15:38:09.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:38:10.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:10 smithi145 ceph-mon[95801]: osdmap e483: 8 total, 8 up, 8 in 2023-07-12T15:38:10.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:10 smithi069 ceph-mon[102777]: osdmap e483: 8 total, 8 up, 8 in 2023-07-12T15:38:10.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:10 smithi069 ceph-mon[94241]: osdmap e483: 8 total, 8 up, 8 in 2023-07-12T15:38:11.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:11 smithi145 ceph-mon[95801]: pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:38:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:11 smithi145 ceph-mon[95801]: osdmap e484: 8 total, 8 up, 8 in 2023-07-12T15:38:11.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:11 smithi069 ceph-mon[102777]: pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:38:11.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:11 smithi069 ceph-mon[102777]: osdmap e484: 8 total, 8 up, 8 in 2023-07-12T15:38:11.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:11 smithi069 ceph-mon[94241]: pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:38:11.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:11 smithi069 ceph-mon[94241]: osdmap e484: 8 total, 8 up, 8 in 2023-07-12T15:38:13.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:12 smithi069 ceph-mon[102777]: pgmap v718: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2023-07-12T15:38:13.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:12 smithi069 ceph-mon[94241]: pgmap v718: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2023-07-12T15:38:13.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:12 smithi145 ceph-mon[95801]: pgmap v718: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2023-07-12T15:38:14.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:38:14.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:38:14.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:38:15.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:14 smithi069 ceph-mon[102777]: pgmap v719: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2023-07-12T15:38:15.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:14 smithi069 ceph-mon[94241]: pgmap v719: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2023-07-12T15:38:15.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:38:14 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:38:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:14 smithi145 ceph-mon[95801]: pgmap v719: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2023-07-12T15:38:16.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:15 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:16.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:15 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:16.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:15 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:16.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:38:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:38:16] "GET /metrics HTTP/1.1" 200 228494 "" "Prometheus/2.43.0" 2023-07-12T15:38:17.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:16 smithi069 ceph-mon[102777]: pgmap v720: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2023-07-12T15:38:17.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:16 smithi069 ceph-mon[94241]: pgmap v720: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2023-07-12T15:38:17.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:16 smithi145 ceph-mon[95801]: pgmap v720: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2023-07-12T15:38:19.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:19 smithi069 ceph-mon[102777]: pgmap v721: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.0 KiB/s wr, 4 op/s 2023-07-12T15:38:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:19 smithi069 ceph-mon[94241]: pgmap v721: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.0 KiB/s wr, 4 op/s 2023-07-12T15:38:19.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:19 smithi145 ceph-mon[95801]: pgmap v721: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.0 KiB/s wr, 4 op/s 2023-07-12T15:38:21.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:21 smithi069 ceph-mon[102777]: pgmap v722: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2023-07-12T15:38:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:21 smithi069 ceph-mon[94241]: pgmap v722: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2023-07-12T15:38:21.605 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:21 smithi145 ceph-mon[95801]: pgmap v722: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2023-07-12T15:38:22.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:22 smithi069 ceph-mon[102777]: osdmap e485: 8 total, 8 up, 8 in 2023-07-12T15:38:22.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:22 smithi069 ceph-mon[94241]: osdmap e485: 8 total, 8 up, 8 in 2023-07-12T15:38:22.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:22 smithi145 ceph-mon[95801]: osdmap e485: 8 total, 8 up, 8 in 2023-07-12T15:38:23.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:23 smithi069 ceph-mon[94241]: pgmap v724: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2023-07-12T15:38:23.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:23 smithi069 ceph-mon[102777]: pgmap v724: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2023-07-12T15:38:23.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:23 smithi145 ceph-mon[95801]: pgmap v724: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2023-07-12T15:38:24.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:24 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:38:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:24 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:38:24.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:24 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:38:25.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:38:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:38:25.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: pgmap v725: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2023-07-12T15:38:25.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: Adjusting osd_memory_target on smithi145 to 3268M 2023-07-12T15:38:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: Adjusting osd_memory_target on smithi069 to 3268M 2023-07-12T15:38:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:38:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:38:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:38:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:38:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:38:25.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: pgmap v725: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2023-07-12T15:38:25.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.490 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.491 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: Adjusting osd_memory_target on smithi145 to 3268M 2023-07-12T15:38:25.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: Adjusting osd_memory_target on smithi069 to 3268M 2023-07-12T15:38:25.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:38:25.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:38:25.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:38:25.492 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:38:25.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:25 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:38:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: pgmap v725: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2023-07-12T15:38:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-12T15:38:25.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: Adjusting osd_memory_target on smithi145 to 3268M 2023-07-12T15:38:25.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: Adjusting osd_memory_target on smithi069 to 3268M 2023-07-12T15:38:25.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:38:25.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:38:25.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:38:25.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:38:25.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:25 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:38:26.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:38:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:38:26] "GET /metrics HTTP/1.1" 200 228494 "" "Prometheus/2.43.0" 2023-07-12T15:38:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:26 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:26 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:26 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:27.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:27 smithi069 ceph-mon[102777]: pgmap v726: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:38:27.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:27 smithi069 ceph-mon[102777]: osdmap e486: 8 total, 8 up, 8 in 2023-07-12T15:38:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:27 smithi069 ceph-mon[94241]: pgmap v726: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:38:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:27 smithi069 ceph-mon[94241]: osdmap e486: 8 total, 8 up, 8 in 2023-07-12T15:38:27.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:27 smithi145 ceph-mon[95801]: pgmap v726: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:38:27.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:27 smithi145 ceph-mon[95801]: osdmap e486: 8 total, 8 up, 8 in 2023-07-12T15:38:29.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:29 smithi069 ceph-mon[102777]: pgmap v728: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:38:29.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:38:29.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:29 smithi069 ceph-mon[94241]: pgmap v728: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:38:29.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:38:29.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:29 smithi145 ceph-mon[95801]: pgmap v728: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:38:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:38:31.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:31 smithi069 ceph-mon[102777]: pgmap v729: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:38:31.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:31 smithi069 ceph-mon[94241]: pgmap v729: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:38:31.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:31 smithi145 ceph-mon[95801]: pgmap v729: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:38:32.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:32.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-94"}]: dispatch 2023-07-12T15:38:32.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:32.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-94"}]: dispatch 2023-07-12T15:38:32.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:32.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-94"}]: dispatch 2023-07-12T15:38:33.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:33 smithi069 ceph-mon[102777]: pgmap v730: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:38:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:33 smithi069 ceph-mon[102777]: osdmap e487: 8 total, 8 up, 8 in 2023-07-12T15:38:33.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:33 smithi069 ceph-mon[94241]: pgmap v730: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:38:33.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:33 smithi069 ceph-mon[94241]: osdmap e487: 8 total, 8 up, 8 in 2023-07-12T15:38:33.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:33 smithi145 ceph-mon[95801]: pgmap v730: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:38:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:33 smithi145 ceph-mon[95801]: osdmap e487: 8 total, 8 up, 8 in 2023-07-12T15:38:34.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:34 smithi145 ceph-mon[95801]: osdmap e488: 8 total, 8 up, 8 in 2023-07-12T15:38:34.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:38:34.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:34 smithi069 ceph-mon[102777]: osdmap e488: 8 total, 8 up, 8 in 2023-07-12T15:38:34.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:38:34.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:34 smithi069 ceph-mon[94241]: osdmap e488: 8 total, 8 up, 8 in 2023-07-12T15:38:34.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:38:35.250 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:38:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:38:35.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:35 smithi145 ceph-mon[95801]: pgmap v733: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:38:35.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-96","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:38:35.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:35 smithi145 ceph-mon[95801]: osdmap e489: 8 total, 8 up, 8 in 2023-07-12T15:38:35.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:38:35.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:35 smithi069 ceph-mon[102777]: pgmap v733: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:38:35.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-96","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:38:35.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:35 smithi069 ceph-mon[102777]: osdmap e489: 8 total, 8 up, 8 in 2023-07-12T15:38:35.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:38:35.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:35 smithi069 ceph-mon[94241]: pgmap v733: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:38:35.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-96","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:38:35.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:35 smithi069 ceph-mon[94241]: osdmap e489: 8 total, 8 up, 8 in 2023-07-12T15:38:35.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:38:36.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:36 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:36.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:36 smithi069 ceph-mon[102777]: osdmap e490: 8 total, 8 up, 8 in 2023-07-12T15:38:36.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:36 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:36.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:36 smithi069 ceph-mon[94241]: osdmap e490: 8 total, 8 up, 8 in 2023-07-12T15:38:36.487 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:38:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:38:36] "GET /metrics HTTP/1.1" 200 228434 "" "Prometheus/2.43.0" 2023-07-12T15:38:36.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:36 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:36.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:36 smithi145 ceph-mon[95801]: osdmap e490: 8 total, 8 up, 8 in 2023-07-12T15:38:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:37 smithi145 ceph-mon[95801]: pgmap v736: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:38:37.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:37 smithi069 ceph-mon[102777]: pgmap v736: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:38:37.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:37 smithi069 ceph-mon[94241]: pgmap v736: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:38:39.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:39 smithi145 ceph-mon[95801]: pgmap v737: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 866 B/s wr, 1 op/s 2023-07-12T15:38:39.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:39 smithi069 ceph-mon[102777]: pgmap v737: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 866 B/s wr, 1 op/s 2023-07-12T15:38:39.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:39 smithi069 ceph-mon[94241]: pgmap v737: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 866 B/s wr, 1 op/s 2023-07-12T15:38:41.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:41 smithi145 ceph-mon[95801]: pgmap v738: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 890 B/s wr, 3 op/s 2023-07-12T15:38:41.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:41 smithi069 ceph-mon[102777]: pgmap v738: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 890 B/s wr, 3 op/s 2023-07-12T15:38:41.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:41 smithi069 ceph-mon[94241]: pgmap v738: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 890 B/s wr, 3 op/s 2023-07-12T15:38:43.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:42 smithi069 ceph-mon[102777]: pgmap v739: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:38:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:42 smithi069 ceph-mon[94241]: pgmap v739: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:38:43.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:42 smithi145 ceph-mon[95801]: pgmap v739: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:38:44.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:43 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:38:44.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:43 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:38:44.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:43 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:38:45.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:44 smithi069 ceph-mon[102777]: pgmap v740: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 620 B/s wr, 2 op/s 2023-07-12T15:38:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:44 smithi069 ceph-mon[94241]: pgmap v740: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 620 B/s wr, 2 op/s 2023-07-12T15:38:45.284 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:38:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:38:45.284 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:44 smithi145 ceph-mon[95801]: pgmap v740: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 620 B/s wr, 2 op/s 2023-07-12T15:38:45.600 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:38:45 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:38:45.281+0000 7f39e58cb700 -1 snap_mapper.add_oid found existing snaps mapped on 62:be88e191:test-rados-api-smithi069-133790-97::foo:31, removing 2023-07-12T15:38:45.601 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:38:45 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:38:45.280+0000 7fa2478bd700 -1 snap_mapper.add_oid found existing snaps mapped on 62:be88e191:test-rados-api-smithi069-133790-97::foo:31, removing 2023-07-12T15:38:45.736 INFO:journalctl@ceph.osd.3.smithi069.stdout:Jul 12 15:38:45 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-3[120895]: 2023-07-12T15:38:45.280+0000 7f9a07dab700 -1 snap_mapper.add_oid found existing snaps mapped on 62:be88e191:test-rados-api-smithi069-133790-97::foo:31, removing 2023-07-12T15:38:46.148 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:45 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:46.148 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:46.148 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-96"}]: dispatch 2023-07-12T15:38:46.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:45 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:46.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:46.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-96"}]: dispatch 2023-07-12T15:38:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:45 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-96"}]: dispatch 2023-07-12T15:38:46.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:38:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:38:46] "GET /metrics HTTP/1.1" 200 228453 "" "Prometheus/2.43.0" 2023-07-12T15:38:47.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:46 smithi069 ceph-mon[102777]: osdmap e491: 8 total, 8 up, 8 in 2023-07-12T15:38:47.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:46 smithi069 ceph-mon[102777]: pgmap v742: 196 pgs: 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:38:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:46 smithi069 ceph-mon[94241]: osdmap e491: 8 total, 8 up, 8 in 2023-07-12T15:38:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:46 smithi069 ceph-mon[94241]: pgmap v742: 196 pgs: 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:38:47.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:46 smithi145 ceph-mon[95801]: osdmap e491: 8 total, 8 up, 8 in 2023-07-12T15:38:47.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:46 smithi145 ceph-mon[95801]: pgmap v742: 196 pgs: 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:38:48.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:47 smithi069 ceph-mon[102777]: osdmap e492: 8 total, 8 up, 8 in 2023-07-12T15:38:48.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:38:48.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:47 smithi069 ceph-mon[94241]: osdmap e492: 8 total, 8 up, 8 in 2023-07-12T15:38:48.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:38:48.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:47 smithi145 ceph-mon[95801]: osdmap e492: 8 total, 8 up, 8 in 2023-07-12T15:38:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:38:49.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-98","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:38:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:48 smithi069 ceph-mon[102777]: osdmap e493: 8 total, 8 up, 8 in 2023-07-12T15:38:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-98", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:38:49.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:48 smithi069 ceph-mon[102777]: pgmap v745: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:38:49.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-98","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:38:49.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:48 smithi069 ceph-mon[94241]: osdmap e493: 8 total, 8 up, 8 in 2023-07-12T15:38:49.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-98", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:38:49.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:48 smithi069 ceph-mon[94241]: pgmap v745: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:38:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-98","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:38:49.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:48 smithi145 ceph-mon[95801]: osdmap e493: 8 total, 8 up, 8 in 2023-07-12T15:38:49.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-98", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:38:49.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:48 smithi145 ceph-mon[95801]: pgmap v745: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:38:50.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-98", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:38:50.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:49 smithi069 ceph-mon[102777]: osdmap e494: 8 total, 8 up, 8 in 2023-07-12T15:38:50.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-98", "mode": "writeback"}]: dispatch 2023-07-12T15:38:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-98", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:38:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:49 smithi069 ceph-mon[94241]: osdmap e494: 8 total, 8 up, 8 in 2023-07-12T15:38:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-98", "mode": "writeback"}]: dispatch 2023-07-12T15:38:50.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-98", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:38:50.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:49 smithi145 ceph-mon[95801]: osdmap e494: 8 total, 8 up, 8 in 2023-07-12T15:38:50.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-98", "mode": "writeback"}]: dispatch 2023-07-12T15:38:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:38:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-98", "mode": "writeback"}]': finished 2023-07-12T15:38:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[102777]: osdmap e495: 8 total, 8 up, 8 in 2023-07-12T15:38:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-98"}]: dispatch 2023-07-12T15:38:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[102777]: pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:38:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-98"}]': finished 2023-07-12T15:38:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[102777]: osdmap e496: 8 total, 8 up, 8 in 2023-07-12T15:38:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:38:51.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:38:51.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-98", "mode": "writeback"}]': finished 2023-07-12T15:38:51.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[94241]: osdmap e495: 8 total, 8 up, 8 in 2023-07-12T15:38:51.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-98"}]: dispatch 2023-07-12T15:38:51.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[94241]: pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:38:51.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-98"}]': finished 2023-07-12T15:38:51.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[94241]: osdmap e496: 8 total, 8 up, 8 in 2023-07-12T15:38:51.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:38:51.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:50 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:38:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-98", "mode": "writeback"}]': finished 2023-07-12T15:38:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:50 smithi145 ceph-mon[95801]: osdmap e495: 8 total, 8 up, 8 in 2023-07-12T15:38:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-98"}]: dispatch 2023-07-12T15:38:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:50 smithi145 ceph-mon[95801]: pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:38:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-98"}]': finished 2023-07-12T15:38:51.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:50 smithi145 ceph-mon[95801]: osdmap e496: 8 total, 8 up, 8 in 2023-07-12T15:38:51.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:38:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:52 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:38:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:38:53.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:52 smithi069 ceph-mon[102777]: osdmap e497: 8 total, 8 up, 8 in 2023-07-12T15:38:53.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_count","val": "1"}]: dispatch 2023-07-12T15:38:53.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:52 smithi069 ceph-mon[102777]: pgmap v751: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:38:53.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:52 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:38:53.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:38:53.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:52 smithi069 ceph-mon[94241]: osdmap e497: 8 total, 8 up, 8 in 2023-07-12T15:38:53.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_count","val": "1"}]: dispatch 2023-07-12T15:38:53.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:52 smithi069 ceph-mon[94241]: pgmap v751: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:38:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:52 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:38:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:38:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:52 smithi145 ceph-mon[95801]: osdmap e497: 8 total, 8 up, 8 in 2023-07-12T15:38:53.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_count","val": "1"}]: dispatch 2023-07-12T15:38:53.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:52 smithi145 ceph-mon[95801]: pgmap v751: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:38:54.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_count","val": "1"}]': finished 2023-07-12T15:38:54.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:53 smithi069 ceph-mon[94241]: osdmap e498: 8 total, 8 up, 8 in 2023-07-12T15:38:54.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:38:54.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_count","val": "1"}]': finished 2023-07-12T15:38:54.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:53 smithi069 ceph-mon[102777]: osdmap e498: 8 total, 8 up, 8 in 2023-07-12T15:38:54.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:38:54.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_count","val": "1"}]': finished 2023-07-12T15:38:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:53 smithi145 ceph-mon[95801]: osdmap e498: 8 total, 8 up, 8 in 2023-07-12T15:38:54.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:38:55.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:38:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:54 smithi069 ceph-mon[102777]: osdmap e499: 8 total, 8 up, 8 in 2023-07-12T15:38:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "target_max_objects","val": "250"}]: dispatch 2023-07-12T15:38:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:54 smithi069 ceph-mon[102777]: pgmap v754: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:38:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:38:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:54 smithi069 ceph-mon[94241]: osdmap e499: 8 total, 8 up, 8 in 2023-07-12T15:38:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "target_max_objects","val": "250"}]: dispatch 2023-07-12T15:38:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:54 smithi069 ceph-mon[94241]: pgmap v754: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:38:55.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:38:54 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:38:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:38:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:54 smithi145 ceph-mon[95801]: osdmap e499: 8 total, 8 up, 8 in 2023-07-12T15:38:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "target_max_objects","val": "250"}]: dispatch 2023-07-12T15:38:55.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:54 smithi145 ceph-mon[95801]: pgmap v754: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:38:56.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:38:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:38:56] "GET /metrics HTTP/1.1" 200 228453 "" "Prometheus/2.43.0" 2023-07-12T15:38:56.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "target_max_objects","val": "250"}]': finished 2023-07-12T15:38:56.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:55 smithi069 ceph-mon[94241]: osdmap e500: 8 total, 8 up, 8 in 2023-07-12T15:38:56.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:56.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:55 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:56.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "target_max_objects","val": "250"}]': finished 2023-07-12T15:38:56.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:55 smithi069 ceph-mon[102777]: osdmap e500: 8 total, 8 up, 8 in 2023-07-12T15:38:56.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:56.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:55 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-98","var": "target_max_objects","val": "250"}]': finished 2023-07-12T15:38:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:55 smithi145 ceph-mon[95801]: osdmap e500: 8 total, 8 up, 8 in 2023-07-12T15:38:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:38:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:55 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:38:57.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:38:57.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:56 smithi069 ceph-mon[94241]: osdmap e501: 8 total, 8 up, 8 in 2023-07-12T15:38:57.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-98"}]: dispatch 2023-07-12T15:38:57.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:56 smithi069 ceph-mon[94241]: pgmap v757: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:38:57.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:38:57.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:56 smithi069 ceph-mon[102777]: osdmap e501: 8 total, 8 up, 8 in 2023-07-12T15:38:57.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-98"}]: dispatch 2023-07-12T15:38:57.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:56 smithi069 ceph-mon[102777]: pgmap v757: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:38:57.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:38:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:56 smithi145 ceph-mon[95801]: osdmap e501: 8 total, 8 up, 8 in 2023-07-12T15:38:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-98"}]: dispatch 2023-07-12T15:38:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:56 smithi145 ceph-mon[95801]: pgmap v757: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:38:58.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-98"}]': finished 2023-07-12T15:38:58.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:57 smithi069 ceph-mon[102777]: osdmap e502: 8 total, 8 up, 8 in 2023-07-12T15:38:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-98"}]': finished 2023-07-12T15:38:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:57 smithi069 ceph-mon[94241]: osdmap e502: 8 total, 8 up, 8 in 2023-07-12T15:38:58.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-98"}]': finished 2023-07-12T15:38:58.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:57 smithi145 ceph-mon[95801]: osdmap e502: 8 total, 8 up, 8 in 2023-07-12T15:38:59.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:58 smithi069 ceph-mon[102777]: osdmap e503: 8 total, 8 up, 8 in 2023-07-12T15:38:59.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:58 smithi069 ceph-mon[102777]: pgmap v760: 196 pgs: 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:38:59.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:58 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:38:59.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:58 smithi069 ceph-mon[94241]: osdmap e503: 8 total, 8 up, 8 in 2023-07-12T15:38:59.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:58 smithi069 ceph-mon[94241]: pgmap v760: 196 pgs: 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:38:59.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:58 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:38:59.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:58 smithi145 ceph-mon[95801]: osdmap e503: 8 total, 8 up, 8 in 2023-07-12T15:38:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:58 smithi145 ceph-mon[95801]: pgmap v760: 196 pgs: 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:38:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:58 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:39:00.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:59 smithi069 ceph-mon[102777]: osdmap e504: 8 total, 8 up, 8 in 2023-07-12T15:39:00.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:38:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:00.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:59 smithi069 ceph-mon[94241]: osdmap e504: 8 total, 8 up, 8 in 2023-07-12T15:39:00.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:38:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:00.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:59 smithi145 ceph-mon[95801]: osdmap e504: 8 total, 8 up, 8 in 2023-07-12T15:39:00.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:38:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:01.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-100","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:01.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:00 smithi069 ceph-mon[94241]: osdmap e505: 8 total, 8 up, 8 in 2023-07-12T15:39:01.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-100", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:39:01.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:00 smithi069 ceph-mon[94241]: pgmap v763: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:01.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-100","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:01.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:00 smithi069 ceph-mon[102777]: osdmap e505: 8 total, 8 up, 8 in 2023-07-12T15:39:01.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-100", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:39:01.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:00 smithi069 ceph-mon[102777]: pgmap v763: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:01.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-100","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:00 smithi145 ceph-mon[95801]: osdmap e505: 8 total, 8 up, 8 in 2023-07-12T15:39:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-100", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:39:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:00 smithi145 ceph-mon[95801]: pgmap v763: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:02.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-100", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:39:02.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:01 smithi145 ceph-mon[95801]: osdmap e506: 8 total, 8 up, 8 in 2023-07-12T15:39:02.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-100"}]: dispatch 2023-07-12T15:39:02.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-100", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:39:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:01 smithi069 ceph-mon[102777]: osdmap e506: 8 total, 8 up, 8 in 2023-07-12T15:39:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-100"}]: dispatch 2023-07-12T15:39:02.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-100", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:39:02.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:01 smithi069 ceph-mon[94241]: osdmap e506: 8 total, 8 up, 8 in 2023-07-12T15:39:02.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-100"}]: dispatch 2023-07-12T15:39:03.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-100"}]': finished 2023-07-12T15:39:03.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:03 smithi145 ceph-mon[95801]: osdmap e507: 8 total, 8 up, 8 in 2023-07-12T15:39:03.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-100", "mode": "writeback"}]: dispatch 2023-07-12T15:39:03.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:03 smithi145 ceph-mon[95801]: pgmap v766: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:03.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:03 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:39:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-100"}]': finished 2023-07-12T15:39:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:03 smithi069 ceph-mon[102777]: osdmap e507: 8 total, 8 up, 8 in 2023-07-12T15:39:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-100", "mode": "writeback"}]: dispatch 2023-07-12T15:39:03.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:03 smithi069 ceph-mon[102777]: pgmap v766: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:03.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:03 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:39:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-6", "overlaypool": "test-rados-api-smithi069-133790-100"}]': finished 2023-07-12T15:39:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:03 smithi069 ceph-mon[94241]: osdmap e507: 8 total, 8 up, 8 in 2023-07-12T15:39:03.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-100", "mode": "writeback"}]: dispatch 2023-07-12T15:39:03.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:03 smithi069 ceph-mon[94241]: pgmap v766: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:03.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:03 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:39:04.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-100", "mode": "writeback"}]': finished 2023-07-12T15:39:04.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:04 smithi145 ceph-mon[95801]: osdmap e508: 8 total, 8 up, 8 in 2023-07-12T15:39:04.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:39:04.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:39:04.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:04 smithi145 ceph-mon[95801]: osdmap e509: 8 total, 8 up, 8 in 2023-07-12T15:39:04.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:39:04.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-100", "mode": "writeback"}]': finished 2023-07-12T15:39:04.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:04 smithi069 ceph-mon[94241]: osdmap e508: 8 total, 8 up, 8 in 2023-07-12T15:39:04.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:39:04.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:39:04.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:04 smithi069 ceph-mon[94241]: osdmap e509: 8 total, 8 up, 8 in 2023-07-12T15:39:04.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:39:04.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-100", "mode": "writeback"}]': finished 2023-07-12T15:39:04.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:04 smithi069 ceph-mon[102777]: osdmap e508: 8 total, 8 up, 8 in 2023-07-12T15:39:04.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:39:04.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:39:04.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:04 smithi069 ceph-mon[102777]: osdmap e509: 8 total, 8 up, 8 in 2023-07-12T15:39:04.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:39:05.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:39:04 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:39:05.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:05 smithi145 ceph-mon[95801]: pgmap v769: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:05.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:39:05.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:05 smithi145 ceph-mon[95801]: osdmap e510: 8 total, 8 up, 8 in 2023-07-12T15:39:05.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:39:05.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:05 smithi069 ceph-mon[102777]: pgmap v769: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:05.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:39:05.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:05 smithi069 ceph-mon[102777]: osdmap e510: 8 total, 8 up, 8 in 2023-07-12T15:39:05.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:39:05.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:05 smithi069 ceph-mon[94241]: pgmap v769: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:05.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:39:05.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:05 smithi069 ceph-mon[94241]: osdmap e510: 8 total, 8 up, 8 in 2023-07-12T15:39:05.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:39:06.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:06 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:06 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:39:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_type","val": "explicit_object"}]': finished 2023-07-12T15:39:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:06 smithi145 ceph-mon[95801]: osdmap e511: 8 total, 8 up, 8 in 2023-07-12T15:39:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2023-07-12T15:39:06.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:39:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:39:06] "GET /metrics HTTP/1.1" 200 226022 "" "Prometheus/2.43.0" 2023-07-12T15:39:06.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:06 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:06.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:06 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:39:06.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_type","val": "explicit_object"}]': finished 2023-07-12T15:39:06.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:06 smithi069 ceph-mon[94241]: osdmap e511: 8 total, 8 up, 8 in 2023-07-12T15:39:06.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2023-07-12T15:39:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:06 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:06 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:39:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "hit_set_type","val": "explicit_object"}]': finished 2023-07-12T15:39:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:06 smithi069 ceph-mon[102777]: osdmap e511: 8 total, 8 up, 8 in 2023-07-12T15:39:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2023-07-12T15:39:07.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:07 smithi145 ceph-mon[95801]: pgmap v772: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2023-07-12T15:39:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:07 smithi145 ceph-mon[95801]: osdmap e512: 8 total, 8 up, 8 in 2023-07-12T15:39:07.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:07 smithi069 ceph-mon[94241]: pgmap v772: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:07.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2023-07-12T15:39:07.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:07 smithi069 ceph-mon[94241]: osdmap e512: 8 total, 8 up, 8 in 2023-07-12T15:39:07.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:07 smithi069 ceph-mon[102777]: pgmap v772: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:07.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2023-07-12T15:39:07.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:07 smithi069 ceph-mon[102777]: osdmap e512: 8 total, 8 up, 8 in 2023-07-12T15:39:08.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:39:08.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:39:08.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:39:09.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:39:09.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:09 smithi145 ceph-mon[95801]: osdmap e513: 8 total, 8 up, 8 in 2023-07-12T15:39:09.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-100"}]: dispatch 2023-07-12T15:39:09.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:09 smithi145 ceph-mon[95801]: pgmap v775: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:09.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:39:09.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:09 smithi069 ceph-mon[102777]: osdmap e513: 8 total, 8 up, 8 in 2023-07-12T15:39:09.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-100"}]: dispatch 2023-07-12T15:39:09.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:09 smithi069 ceph-mon[102777]: pgmap v775: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:09.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]': finished 2023-07-12T15:39:09.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:09 smithi069 ceph-mon[94241]: osdmap e513: 8 total, 8 up, 8 in 2023-07-12T15:39:09.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-100"}]: dispatch 2023-07-12T15:39:09.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:09 smithi069 ceph-mon[94241]: pgmap v775: 228 pgs: 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:10.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-100"}]': finished 2023-07-12T15:39:10.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:10 smithi145 ceph-mon[95801]: osdmap e514: 8 total, 8 up, 8 in 2023-07-12T15:39:10.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-100"}]': finished 2023-07-12T15:39:10.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:10 smithi069 ceph-mon[102777]: osdmap e514: 8 total, 8 up, 8 in 2023-07-12T15:39:10.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-100"}]': finished 2023-07-12T15:39:10.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:10 smithi069 ceph-mon[94241]: osdmap e514: 8 total, 8 up, 8 in 2023-07-12T15:39:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:11 smithi145 ceph-mon[95801]: osdmap e515: 8 total, 8 up, 8 in 2023-07-12T15:39:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:11 smithi145 ceph-mon[95801]: pgmap v778: 196 pgs: 196 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:39:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:11 smithi145 ceph-mon[95801]: osdmap e516: 8 total, 8 up, 8 in 2023-07-12T15:39:11.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:11.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:11 smithi069 ceph-mon[102777]: osdmap e515: 8 total, 8 up, 8 in 2023-07-12T15:39:11.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:11 smithi069 ceph-mon[102777]: pgmap v778: 196 pgs: 196 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:39:11.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:11 smithi069 ceph-mon[102777]: osdmap e516: 8 total, 8 up, 8 in 2023-07-12T15:39:11.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:11.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:11 smithi069 ceph-mon[94241]: osdmap e515: 8 total, 8 up, 8 in 2023-07-12T15:39:11.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:11 smithi069 ceph-mon[94241]: pgmap v778: 196 pgs: 196 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:39:11.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:11 smithi069 ceph-mon[94241]: osdmap e516: 8 total, 8 up, 8 in 2023-07-12T15:39:11.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:13.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-102","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:13 smithi145 ceph-mon[95801]: osdmap e517: 8 total, 8 up, 8 in 2023-07-12T15:39:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:39:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:39:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:13 smithi145 ceph-mon[95801]: pgmap v781: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:39:13.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-102","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:13.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:13 smithi069 ceph-mon[94241]: osdmap e517: 8 total, 8 up, 8 in 2023-07-12T15:39:13.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:39:13.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:39:13.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:13 smithi069 ceph-mon[94241]: pgmap v781: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:39:13.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-102","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:13.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:13 smithi069 ceph-mon[102777]: osdmap e517: 8 total, 8 up, 8 in 2023-07-12T15:39:13.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2023-07-12T15:39:13.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2023-07-12T15:39:13.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:13 smithi069 ceph-mon[102777]: pgmap v781: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:39:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:39:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:14 smithi145 ceph-mon[95801]: osdmap e518: 8 total, 8 up, 8 in 2023-07-12T15:39:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:39:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:14 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:39:14.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:39:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:14 smithi069 ceph-mon[102777]: osdmap e518: 8 total, 8 up, 8 in 2023-07-12T15:39:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:39:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:14 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:39:14.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2023-07-12T15:39:14.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:14 smithi069 ceph-mon[94241]: osdmap e518: 8 total, 8 up, 8 in 2023-07-12T15:39:14.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2023-07-12T15:39:14.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:14 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:39:15.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:39:14 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:39:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:39:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:15 smithi145 ceph-mon[95801]: osdmap e519: 8 total, 8 up, 8 in 2023-07-12T15:39:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:39:15.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:15 smithi145 ceph-mon[95801]: pgmap v784: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:39:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:15 smithi069 ceph-mon[102777]: osdmap e519: 8 total, 8 up, 8 in 2023-07-12T15:39:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:39:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:15 smithi069 ceph-mon[102777]: pgmap v784: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:15.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2023-07-12T15:39:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:15 smithi069 ceph-mon[94241]: osdmap e519: 8 total, 8 up, 8 in 2023-07-12T15:39:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2023-07-12T15:39:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:15 smithi069 ceph-mon[94241]: pgmap v784: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:16.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:16 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:39:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:16 smithi145 ceph-mon[95801]: osdmap e520: 8 total, 8 up, 8 in 2023-07-12T15:39:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:39:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-102"}]: dispatch 2023-07-12T15:39:16.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:39:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:39:16] "GET /metrics HTTP/1.1" 200 228467 "" "Prometheus/2.43.0" 2023-07-12T15:39:16.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:16 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:16.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:39:16.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:16 smithi069 ceph-mon[102777]: osdmap e520: 8 total, 8 up, 8 in 2023-07-12T15:39:16.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:39:16.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-102"}]: dispatch 2023-07-12T15:39:16.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:16 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:16.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2023-07-12T15:39:16.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:16 smithi069 ceph-mon[94241]: osdmap e520: 8 total, 8 up, 8 in 2023-07-12T15:39:16.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-6"}]: dispatch 2023-07-12T15:39:16.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1905028013' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-6", "tierpool": "test-rados-api-smithi069-133790-102"}]: dispatch 2023-07-12T15:39:17.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:17 smithi145 ceph-mon[95801]: osdmap e521: 8 total, 8 up, 8 in 2023-07-12T15:39:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:17 smithi145 ceph-mon[95801]: pgmap v787: 196 pgs: 196 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:17.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:17 smithi069 ceph-mon[102777]: osdmap e521: 8 total, 8 up, 8 in 2023-07-12T15:39:17.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:17 smithi069 ceph-mon[102777]: pgmap v787: 196 pgs: 196 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:17.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:17 smithi069 ceph-mon[94241]: osdmap e521: 8 total, 8 up, 8 in 2023-07-12T15:39:17.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:17 smithi069 ceph-mon[94241]: pgmap v787: 196 pgs: 196 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:18 smithi145 ceph-mon[95801]: osdmap e522: 8 total, 8 up, 8 in 2023-07-12T15:39:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:18 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:18 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:18.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:18.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:18 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:18.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[102777]: osdmap e522: 8 total, 8 up, 8 in 2023-07-12T15:39:18.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:18.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:18.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:18.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:18.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:18.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[94241]: osdmap e522: 8 total, 8 up, 8 in 2023-07-12T15:39:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:18.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:18 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:19.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:19 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:39:19.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:19 smithi069 ceph-mon[102777]: osdmap e523: 8 total, 8 up, 8 in 2023-07-12T15:39:19.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-133790-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:19.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:19 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-133790-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:19.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:19 smithi069 ceph-mon[102777]: pgmap v790: 164 pgs: 164 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:19 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:39:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:19 smithi069 ceph-mon[94241]: osdmap e523: 8 total, 8 up, 8 in 2023-07-12T15:39:19.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-133790-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:19.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:19 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-133790-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:19.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:19 smithi069 ceph-mon[94241]: pgmap v790: 164 pgs: 164 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:19.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:19 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:39:19.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:19 smithi145 ceph-mon[95801]: osdmap e523: 8 total, 8 up, 8 in 2023-07-12T15:39:19.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-133790-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:19.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:19 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-133790-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:19.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:19 smithi145 ceph-mon[95801]: pgmap v790: 164 pgs: 164 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:20.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:20 smithi069 ceph-mon[102777]: osdmap e524: 8 total, 8 up, 8 in 2023-07-12T15:39:20.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:20 smithi069 ceph-mon[94241]: osdmap e524: 8 total, 8 up, 8 in 2023-07-12T15:39:20.605 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:20 smithi145 ceph-mon[95801]: osdmap e524: 8 total, 8 up, 8 in 2023-07-12T15:39:21.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:21 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-133790-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-133790-104"}]': finished 2023-07-12T15:39:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:21 smithi069 ceph-mon[94241]: osdmap e525: 8 total, 8 up, 8 in 2023-07-12T15:39:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:21 smithi069 ceph-mon[94241]: pgmap v793: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:21 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-133790-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-133790-104"}]': finished 2023-07-12T15:39:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:21 smithi069 ceph-mon[102777]: osdmap e525: 8 total, 8 up, 8 in 2023-07-12T15:39:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:21 smithi069 ceph-mon[102777]: pgmap v793: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:21 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi069-133790-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi069-133790-104"}]': finished 2023-07-12T15:39:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:21 smithi145 ceph-mon[95801]: osdmap e525: 8 total, 8 up, 8 in 2023-07-12T15:39:21.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:21 smithi145 ceph-mon[95801]: pgmap v793: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:22.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:22 smithi069 ceph-mon[94241]: osdmap e526: 8 total, 8 up, 8 in 2023-07-12T15:39:22.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:22 smithi069 ceph-mon[102777]: osdmap e526: 8 total, 8 up, 8 in 2023-07-12T15:39:22.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:22 smithi145 ceph-mon[95801]: osdmap e526: 8 total, 8 up, 8 in 2023-07-12T15:39:23.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:23 smithi069 ceph-mon[102777]: osdmap e527: 8 total, 8 up, 8 in 2023-07-12T15:39:23.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:23.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:23 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:23.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:23 smithi069 ceph-mon[102777]: pgmap v796: 204 pgs: 40 unknown, 164 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:23 smithi069 ceph-mon[94241]: osdmap e527: 8 total, 8 up, 8 in 2023-07-12T15:39:23.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:23.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:23 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:23.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:23 smithi069 ceph-mon[94241]: pgmap v796: 204 pgs: 40 unknown, 164 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:23 smithi145 ceph-mon[95801]: osdmap e527: 8 total, 8 up, 8 in 2023-07-12T15:39:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:23.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:23 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:23.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:23 smithi145 ceph-mon[95801]: pgmap v796: 204 pgs: 40 unknown, 164 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:24.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:24 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:24.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:24 smithi069 ceph-mon[102777]: osdmap e528: 8 total, 8 up, 8 in 2023-07-12T15:39:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:24 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:24 smithi069 ceph-mon[94241]: osdmap e528: 8 total, 8 up, 8 in 2023-07-12T15:39:24.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:24 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:24.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:24 smithi145 ceph-mon[95801]: osdmap e528: 8 total, 8 up, 8 in 2023-07-12T15:39:25.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:39:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:39:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:25 smithi145 ceph-mon[95801]: osdmap e529: 8 total, 8 up, 8 in 2023-07-12T15:39:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:25.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:25 smithi145 ceph-mon[95801]: pgmap v799: 236 pgs: 72 unknown, 164 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:25.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:39:25.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:39:25.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:39:25.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:25 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:39:25.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:25 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:25.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:25 smithi145 ceph-mon[95801]: osdmap e530: 8 total, 8 up, 8 in 2023-07-12T15:39:25.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[94241]: osdmap e529: 8 total, 8 up, 8 in 2023-07-12T15:39:25.499 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:25.499 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:25.499 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[94241]: pgmap v799: 236 pgs: 72 unknown, 164 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:25.499 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:39:25.499 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:39:25.500 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:39:25.500 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:39:25.500 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:25.500 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[94241]: osdmap e530: 8 total, 8 up, 8 in 2023-07-12T15:39:25.500 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[102777]: osdmap e529: 8 total, 8 up, 8 in 2023-07-12T15:39:25.501 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:25.501 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:25.501 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[102777]: pgmap v799: 236 pgs: 72 unknown, 164 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:25.501 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:39:25.501 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:39:25.501 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:39:25.501 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:39:25.502 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:25.502 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:25 smithi069 ceph-mon[102777]: osdmap e530: 8 total, 8 up, 8 in 2023-07-12T15:39:26.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:39:26] "GET /metrics HTTP/1.1" 200 228467 "" "Prometheus/2.43.0" 2023-07-12T15:39:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:26.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:26.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]': finished 2023-07-12T15:39:26.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[102777]: osdmap e531: 8 total, 8 up, 8 in 2023-07-12T15:39:26.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-107", "overlaypool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:26.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-107", "overlaypool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]': finished 2023-07-12T15:39:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[94241]: osdmap e531: 8 total, 8 up, 8 in 2023-07-12T15:39:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-107", "overlaypool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:26 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-107", "overlaypool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:26.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:26 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]': finished 2023-07-12T15:39:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:26 smithi145 ceph-mon[95801]: osdmap e531: 8 total, 8 up, 8 in 2023-07-12T15:39:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-107", "overlaypool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:26 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-107", "overlaypool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:27 smithi069 ceph-mon[94241]: pgmap v802: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:39:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:27 smithi069 ceph-mon[94241]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:39:27.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:27 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-107", "overlaypool": "test-rados-api-smithi069-133790-107-cache"}]': finished 2023-07-12T15:39:27.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:27 smithi069 ceph-mon[94241]: osdmap e532: 8 total, 8 up, 8 in 2023-07-12T15:39:27.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-107-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:39:27.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:27 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-107-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:39:27.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:27 smithi069 ceph-mon[102777]: pgmap v802: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:39:27.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:27 smithi069 ceph-mon[102777]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:39:27.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:27 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-107", "overlaypool": "test-rados-api-smithi069-133790-107-cache"}]': finished 2023-07-12T15:39:27.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:27 smithi069 ceph-mon[102777]: osdmap e532: 8 total, 8 up, 8 in 2023-07-12T15:39:27.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-107-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:39:27.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:27 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-107-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:39:27.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:27 smithi145 ceph-mon[95801]: pgmap v802: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:39:27.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:27 smithi145 ceph-mon[95801]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:39:27.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:27 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-107", "overlaypool": "test-rados-api-smithi069-133790-107-cache"}]': finished 2023-07-12T15:39:27.603 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:27 smithi145 ceph-mon[95801]: osdmap e532: 8 total, 8 up, 8 in 2023-07-12T15:39:27.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-107-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:39:27.604 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:27 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-107-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:39:28.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:28 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:39:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:28 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-107-cache", "mode": "writeback"}]': finished 2023-07-12T15:39:28.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:28 smithi069 ceph-mon[102777]: osdmap e533: 8 total, 8 up, 8 in 2023-07-12T15:39:28.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:28 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:39:28.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:28 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-107-cache", "mode": "writeback"}]': finished 2023-07-12T15:39:28.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:28 smithi069 ceph-mon[94241]: osdmap e533: 8 total, 8 up, 8 in 2023-07-12T15:39:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:28 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:39:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:28 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-107-cache", "mode": "writeback"}]': finished 2023-07-12T15:39:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:28 smithi145 ceph-mon[95801]: osdmap e533: 8 total, 8 up, 8 in 2023-07-12T15:39:29.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:29 smithi069 ceph-mon[102777]: pgmap v804: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:39:29.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:29 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-107"}]: dispatch 2023-07-12T15:39:29.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:29 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-107"}]: dispatch 2023-07-12T15:39:29.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:29 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:39:29.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:39:29.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:29 smithi069 ceph-mon[94241]: pgmap v804: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:39:29.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:29 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-107"}]: dispatch 2023-07-12T15:39:29.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:29 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-107"}]: dispatch 2023-07-12T15:39:29.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:29 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:39:29.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:39:29.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:29 smithi145 ceph-mon[95801]: pgmap v804: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:39:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:29 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-107"}]: dispatch 2023-07-12T15:39:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:29 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-107"}]: dispatch 2023-07-12T15:39:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:29 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:39:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:39:30.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:30 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-107"}]': finished 2023-07-12T15:39:30.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:30 smithi069 ceph-mon[102777]: osdmap e534: 8 total, 8 up, 8 in 2023-07-12T15:39:30.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:30.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:30 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:30.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:30 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-107"}]': finished 2023-07-12T15:39:30.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:30 smithi069 ceph-mon[94241]: osdmap e534: 8 total, 8 up, 8 in 2023-07-12T15:39:30.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:30.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:30 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:30 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-107"}]': finished 2023-07-12T15:39:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:30 smithi145 ceph-mon[95801]: osdmap e534: 8 total, 8 up, 8 in 2023-07-12T15:39:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3602672599' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:30.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:30 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]: dispatch 2023-07-12T15:39:31.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:31 smithi069 ceph-mon[102777]: pgmap v807: 236 pgs: 236 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:39:31.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:31 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:39:31.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:31 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]': finished 2023-07-12T15:39:31.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:31 smithi069 ceph-mon[102777]: osdmap e535: 8 total, 8 up, 8 in 2023-07-12T15:39:31.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:31 smithi069 ceph-mon[94241]: pgmap v807: 236 pgs: 236 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:39:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:31 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:39:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:31 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]': finished 2023-07-12T15:39:31.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:31 smithi069 ceph-mon[94241]: osdmap e535: 8 total, 8 up, 8 in 2023-07-12T15:39:31.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:31 smithi145 ceph-mon[95801]: pgmap v807: 236 pgs: 236 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:39:31.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:31 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:39:31.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:31 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-107", "tierpool": "test-rados-api-smithi069-133790-107-cache"}]': finished 2023-07-12T15:39:31.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:31 smithi145 ceph-mon[95801]: osdmap e535: 8 total, 8 up, 8 in 2023-07-12T15:39:32.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:32 smithi069 ceph-mon[102777]: osdmap e536: 8 total, 8 up, 8 in 2023-07-12T15:39:32.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:32 smithi069 ceph-mon[94241]: osdmap e536: 8 total, 8 up, 8 in 2023-07-12T15:39:32.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:32 smithi145 ceph-mon[95801]: osdmap e536: 8 total, 8 up, 8 in 2023-07-12T15:39:33.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[94241]: pgmap v810: 204 pgs: 204 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:39:33.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[94241]: osdmap e537: 8 total, 8 up, 8 in 2023-07-12T15:39:33.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:33.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:33.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:33.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[102777]: pgmap v810: 204 pgs: 204 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:39:33.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[102777]: osdmap e537: 8 total, 8 up, 8 in 2023-07-12T15:39:33.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:33.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:33.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:33.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:33.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:33.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:33 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:33.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:33 smithi145 ceph-mon[95801]: pgmap v810: 204 pgs: 204 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:39:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:33 smithi145 ceph-mon[95801]: osdmap e537: 8 total, 8 up, 8 in 2023-07-12T15:39:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:33 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:33 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:33.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:33.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:33 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:34.218 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:39:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=sqlstore.transactions t=2023-07-12T15:39:33.908330541Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2023-07-12T15:39:34.218 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:39:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=cleanup t=2023-07-12T15:39:33.910077398Z level=info msg="Completed cleanup jobs" duration=13.18134ms 2023-07-12T15:39:34.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:34 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:39:34.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:34 smithi069 ceph-mon[102777]: osdmap e538: 8 total, 8 up, 8 in 2023-07-12T15:39:34.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:34.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:34 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:34.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:34 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:39:34.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:34 smithi069 ceph-mon[94241]: osdmap e538: 8 total, 8 up, 8 in 2023-07-12T15:39:34.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:34.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:34 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:34.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:34 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:39:34.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:34 smithi145 ceph-mon[95801]: osdmap e538: 8 total, 8 up, 8 in 2023-07-12T15:39:34.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:34.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:34 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:35.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:39:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:39:35.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:35 smithi145 ceph-mon[95801]: pgmap v813: 172 pgs: 172 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:35.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:35 smithi145 ceph-mon[95801]: osdmap e539: 8 total, 8 up, 8 in 2023-07-12T15:39:35.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:35 smithi069 ceph-mon[102777]: pgmap v813: 172 pgs: 172 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:35.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:35 smithi069 ceph-mon[102777]: osdmap e539: 8 total, 8 up, 8 in 2023-07-12T15:39:35.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:35 smithi069 ceph-mon[94241]: pgmap v813: 172 pgs: 172 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:35.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:35 smithi069 ceph-mon[94241]: osdmap e539: 8 total, 8 up, 8 in 2023-07-12T15:39:36.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:39:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:39:36] "GET /metrics HTTP/1.1" 200 230908 "" "Prometheus/2.43.0" 2023-07-12T15:39:36.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:36 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:36.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:36 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-109"}]': finished 2023-07-12T15:39:36.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:36 smithi069 ceph-mon[102777]: osdmap e540: 8 total, 8 up, 8 in 2023-07-12T15:39:36.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:36 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:36.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:36 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-109"}]': finished 2023-07-12T15:39:36.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:36 smithi069 ceph-mon[94241]: osdmap e540: 8 total, 8 up, 8 in 2023-07-12T15:39:36.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:36 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:36.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:36 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-109"}]': finished 2023-07-12T15:39:36.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:36 smithi145 ceph-mon[95801]: osdmap e540: 8 total, 8 up, 8 in 2023-07-12T15:39:37.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:37 smithi069 ceph-mon[102777]: pgmap v816: 180 pgs: 8 unknown, 172 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:37.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:37 smithi069 ceph-mon[102777]: osdmap e541: 8 total, 8 up, 8 in 2023-07-12T15:39:37.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:37.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:37 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:37.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:37 smithi069 ceph-mon[94241]: pgmap v816: 180 pgs: 8 unknown, 172 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:37.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:37 smithi069 ceph-mon[94241]: osdmap e541: 8 total, 8 up, 8 in 2023-07-12T15:39:37.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:37.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:37 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:37.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:37 smithi145 ceph-mon[95801]: pgmap v816: 180 pgs: 8 unknown, 172 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:37 smithi145 ceph-mon[95801]: osdmap e541: 8 total, 8 up, 8 in 2023-07-12T15:39:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:37 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:38.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:38 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:38.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:38 smithi145 ceph-mon[95801]: osdmap e542: 8 total, 8 up, 8 in 2023-07-12T15:39:38.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:38.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:38 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:38.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:38 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:38.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:38 smithi069 ceph-mon[94241]: osdmap e542: 8 total, 8 up, 8 in 2023-07-12T15:39:38.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:38.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:38 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:38.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:38 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:38.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:38 smithi069 ceph-mon[102777]: osdmap e542: 8 total, 8 up, 8 in 2023-07-12T15:39:38.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:38.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:38 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:39.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:39 smithi145 ceph-mon[95801]: pgmap v819: 212 pgs: 40 unknown, 172 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:39.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:39 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]': finished 2023-07-12T15:39:39.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:39 smithi145 ceph-mon[95801]: osdmap e543: 8 total, 8 up, 8 in 2023-07-12T15:39:39.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-109", "overlaypool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:39.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:39 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-109", "overlaypool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:39.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:39 smithi069 ceph-mon[94241]: pgmap v819: 212 pgs: 40 unknown, 172 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:39.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:39 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]': finished 2023-07-12T15:39:39.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:39 smithi069 ceph-mon[94241]: osdmap e543: 8 total, 8 up, 8 in 2023-07-12T15:39:39.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-109", "overlaypool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:39.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:39 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-109", "overlaypool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:39.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:39 smithi069 ceph-mon[102777]: pgmap v819: 212 pgs: 40 unknown, 172 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:39.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:39 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]': finished 2023-07-12T15:39:39.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:39 smithi069 ceph-mon[102777]: osdmap e543: 8 total, 8 up, 8 in 2023-07-12T15:39:39.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-109", "overlaypool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:39.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:39 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-109", "overlaypool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:40.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:40 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-109", "overlaypool": "test-rados-api-smithi069-133790-109-cache"}]': finished 2023-07-12T15:39:40.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:40 smithi145 ceph-mon[95801]: osdmap e544: 8 total, 8 up, 8 in 2023-07-12T15:39:40.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-109-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:39:40.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:40 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-109-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:39:40.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:40 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:39:40.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:40 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-109", "overlaypool": "test-rados-api-smithi069-133790-109-cache"}]': finished 2023-07-12T15:39:40.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:40 smithi069 ceph-mon[94241]: osdmap e544: 8 total, 8 up, 8 in 2023-07-12T15:39:40.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-109-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:39:40.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:40 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-109-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:39:40.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:40 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:39:40.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:40 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-109", "overlaypool": "test-rados-api-smithi069-133790-109-cache"}]': finished 2023-07-12T15:39:40.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:40 smithi069 ceph-mon[102777]: osdmap e544: 8 total, 8 up, 8 in 2023-07-12T15:39:40.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-109-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:39:40.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:40 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-109-cache", "mode": "writeback"}]: dispatch 2023-07-12T15:39:40.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:40 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:39:41.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:41 smithi145 ceph-mon[95801]: pgmap v822: 212 pgs: 212 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:41.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:41 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-109-cache", "mode": "writeback"}]': finished 2023-07-12T15:39:41.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:41 smithi145 ceph-mon[95801]: osdmap e545: 8 total, 8 up, 8 in 2023-07-12T15:39:41.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:39:41.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:41 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:39:41.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:41 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:39:41.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:41 smithi145 ceph-mon[95801]: osdmap e546: 8 total, 8 up, 8 in 2023-07-12T15:39:41.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:39:41.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:41 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:39:41.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[94241]: pgmap v822: 212 pgs: 212 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:41.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-109-cache", "mode": "writeback"}]': finished 2023-07-12T15:39:41.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[94241]: osdmap e545: 8 total, 8 up, 8 in 2023-07-12T15:39:41.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:39:41.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:39:41.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:39:41.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[94241]: osdmap e546: 8 total, 8 up, 8 in 2023-07-12T15:39:41.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:39:41.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:39:41.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[102777]: pgmap v822: 212 pgs: 212 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:41.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-109-cache", "mode": "writeback"}]': finished 2023-07-12T15:39:41.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[102777]: osdmap e545: 8 total, 8 up, 8 in 2023-07-12T15:39:41.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:39:41.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:39:41.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:39:41.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[102777]: osdmap e546: 8 total, 8 up, 8 in 2023-07-12T15:39:41.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:39:41.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:41 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:39:43.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:43 smithi145 ceph-mon[95801]: pgmap v825: 212 pgs: 212 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:43 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:39:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:43 smithi145 ceph-mon[95801]: osdmap e547: 8 total, 8 up, 8 in 2023-07-12T15:39:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:39:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:43 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:39:43.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:43 smithi069 ceph-mon[94241]: pgmap v825: 212 pgs: 212 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:43.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:43 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:39:43.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:43 smithi069 ceph-mon[94241]: osdmap e547: 8 total, 8 up, 8 in 2023-07-12T15:39:43.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:39:43.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:43 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:39:43.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:43 smithi069 ceph-mon[102777]: pgmap v825: 212 pgs: 212 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:43.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:43 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:39:43.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:43 smithi069 ceph-mon[102777]: osdmap e547: 8 total, 8 up, 8 in 2023-07-12T15:39:43.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:39:43.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:43 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:39:44.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:44 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:39:44.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:44 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2023-07-12T15:39:44.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:44 smithi145 ceph-mon[95801]: osdmap e548: 8 total, 8 up, 8 in 2023-07-12T15:39:44.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2023-07-12T15:39:44.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:44 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2023-07-12T15:39:44.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:44 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:39:44.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:44 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:39:44.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:39:44.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2023-07-12T15:39:44.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[94241]: osdmap e548: 8 total, 8 up, 8 in 2023-07-12T15:39:44.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2023-07-12T15:39:44.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2023-07-12T15:39:44.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:39:44.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:39:44.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:39:44.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2023-07-12T15:39:44.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[102777]: osdmap e548: 8 total, 8 up, 8 in 2023-07-12T15:39:44.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2023-07-12T15:39:44.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2023-07-12T15:39:44.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:39:44.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:44 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:39:45.289 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:39:45 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:39:45.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:45 smithi145 ceph-mon[95801]: pgmap v828: 212 pgs: 212 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:45.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:45 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2023-07-12T15:39:45.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:45 smithi145 ceph-mon[95801]: osdmap e549: 8 total, 8 up, 8 in 2023-07-12T15:39:45.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:45.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:45 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:45.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:45 smithi069 ceph-mon[102777]: pgmap v828: 212 pgs: 212 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:45.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:45 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2023-07-12T15:39:45.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:45 smithi069 ceph-mon[102777]: osdmap e549: 8 total, 8 up, 8 in 2023-07-12T15:39:45.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:45.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:45 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:45.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:45 smithi069 ceph-mon[94241]: pgmap v828: 212 pgs: 212 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:45.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:45 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2023-07-12T15:39:45.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:45 smithi069 ceph-mon[94241]: osdmap e549: 8 total, 8 up, 8 in 2023-07-12T15:39:45.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:45.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:45 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:46.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:39:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:39:46] "GET /metrics HTTP/1.1" 200 230881 "" "Prometheus/2.43.0" 2023-07-12T15:39:46.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:46 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-109"}]': finished 2023-07-12T15:39:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:46 smithi069 ceph-mon[102777]: osdmap e550: 8 total, 8 up, 8 in 2023-07-12T15:39:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:46 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]': finished 2023-07-12T15:39:46.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:46 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:46.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-109"}]': finished 2023-07-12T15:39:46.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:46 smithi069 ceph-mon[94241]: osdmap e550: 8 total, 8 up, 8 in 2023-07-12T15:39:46.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:46.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:46.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:46 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]': finished 2023-07-12T15:39:46.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:46 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:46.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-109"}]': finished 2023-07-12T15:39:46.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:46 smithi145 ceph-mon[95801]: osdmap e550: 8 total, 8 up, 8 in 2023-07-12T15:39:46.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:46.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]: dispatch 2023-07-12T15:39:46.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:46 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-109", "tierpool": "test-rados-api-smithi069-133790-109-cache"}]': finished 2023-07-12T15:39:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:47 smithi145 ceph-mon[95801]: pgmap v831: 212 pgs: 212 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:39:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:47 smithi145 ceph-mon[95801]: osdmap e551: 8 total, 8 up, 8 in 2023-07-12T15:39:47.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:47 smithi145 ceph-mon[95801]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:39:47.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:47 smithi069 ceph-mon[102777]: pgmap v831: 212 pgs: 212 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:39:47.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:47 smithi069 ceph-mon[102777]: osdmap e551: 8 total, 8 up, 8 in 2023-07-12T15:39:47.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:47 smithi069 ceph-mon[102777]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:39:47.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:47 smithi069 ceph-mon[94241]: pgmap v831: 212 pgs: 212 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:39:47.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:47 smithi069 ceph-mon[94241]: osdmap e551: 8 total, 8 up, 8 in 2023-07-12T15:39:47.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:47 smithi069 ceph-mon[94241]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:39:48.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:48 smithi145 ceph-mon[95801]: osdmap e552: 8 total, 8 up, 8 in 2023-07-12T15:39:48.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:48 smithi069 ceph-mon[102777]: osdmap e552: 8 total, 8 up, 8 in 2023-07-12T15:39:48.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:48 smithi069 ceph-mon[94241]: osdmap e552: 8 total, 8 up, 8 in 2023-07-12T15:39:49.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:49 smithi145 ceph-mon[95801]: pgmap v834: 180 pgs: 180 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:39:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:49 smithi145 ceph-mon[95801]: osdmap e553: 8 total, 8 up, 8 in 2023-07-12T15:39:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:49 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:49.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:49 smithi069 ceph-mon[102777]: pgmap v834: 180 pgs: 180 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:39:49.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:49 smithi069 ceph-mon[102777]: osdmap e553: 8 total, 8 up, 8 in 2023-07-12T15:39:49.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:49.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:49 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:49.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:49 smithi069 ceph-mon[94241]: pgmap v834: 180 pgs: 180 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:39:49.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:49 smithi069 ceph-mon[94241]: osdmap e553: 8 total, 8 up, 8 in 2023-07-12T15:39:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:49 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:50.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]': finished 2023-07-12T15:39:50.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:50 smithi145 ceph-mon[95801]: osdmap e554: 8 total, 8 up, 8 in 2023-07-12T15:39:50.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:50.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:50 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:50.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]': finished 2023-07-12T15:39:50.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:50 smithi069 ceph-mon[102777]: osdmap e554: 8 total, 8 up, 8 in 2023-07-12T15:39:50.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:50.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:50 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:50.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-109"}]': finished 2023-07-12T15:39:50.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:50 smithi069 ceph-mon[94241]: osdmap e554: 8 total, 8 up, 8 in 2023-07-12T15:39:50.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2697317395' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:50.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:50 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]: dispatch 2023-07-12T15:39:51.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:51 smithi145 ceph-mon[95801]: pgmap v837: 172 pgs: 172 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2023-07-12T15:39:51.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]': finished 2023-07-12T15:39:51.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:51 smithi145 ceph-mon[95801]: osdmap e555: 8 total, 8 up, 8 in 2023-07-12T15:39:51.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:51 smithi069 ceph-mon[102777]: pgmap v837: 172 pgs: 172 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2023-07-12T15:39:51.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]': finished 2023-07-12T15:39:51.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:51 smithi069 ceph-mon[102777]: osdmap e555: 8 total, 8 up, 8 in 2023-07-12T15:39:51.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:51 smithi069 ceph-mon[94241]: pgmap v837: 172 pgs: 172 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2023-07-12T15:39:51.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-109"}]': finished 2023-07-12T15:39:51.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:51 smithi069 ceph-mon[94241]: osdmap e555: 8 total, 8 up, 8 in 2023-07-12T15:39:52.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:52 smithi145 ceph-mon[95801]: osdmap e556: 8 total, 8 up, 8 in 2023-07-12T15:39:52.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:52.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:52.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:52 smithi145 ceph-mon[95801]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:39:52.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:52 smithi069 ceph-mon[102777]: osdmap e556: 8 total, 8 up, 8 in 2023-07-12T15:39:52.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:52.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:52.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:52 smithi069 ceph-mon[102777]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:39:52.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:52 smithi069 ceph-mon[94241]: osdmap e556: 8 total, 8 up, 8 in 2023-07-12T15:39:52.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:52.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:52.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:52 smithi069 ceph-mon[94241]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:39:53.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:53 smithi145 ceph-mon[95801]: pgmap v840: 164 pgs: 164 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2023-07-12T15:39:53.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]': finished 2023-07-12T15:39:53.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:53 smithi145 ceph-mon[95801]: osdmap e557: 8 total, 8 up, 8 in 2023-07-12T15:39:53.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:53.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:53.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:53 smithi069 ceph-mon[102777]: pgmap v840: 164 pgs: 164 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2023-07-12T15:39:53.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]': finished 2023-07-12T15:39:53.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:53 smithi069 ceph-mon[102777]: osdmap e557: 8 total, 8 up, 8 in 2023-07-12T15:39:53.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:53.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:53.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:53 smithi069 ceph-mon[94241]: pgmap v840: 164 pgs: 164 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2023-07-12T15:39:53.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi069-133790-104"}]': finished 2023-07-12T15:39:53.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:53 smithi069 ceph-mon[94241]: osdmap e557: 8 total, 8 up, 8 in 2023-07-12T15:39:53.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/4043312419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:53.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]: dispatch 2023-07-12T15:39:54.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]': finished 2023-07-12T15:39:54.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:54 smithi069 ceph-mon[94241]: osdmap e558: 8 total, 8 up, 8 in 2023-07-12T15:39:54.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:39:54.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:39:54.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:54.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]': finished 2023-07-12T15:39:54.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:54 smithi069 ceph-mon[102777]: osdmap e558: 8 total, 8 up, 8 in 2023-07-12T15:39:54.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:39:54.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:39:54.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:54.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi069-133790-104"}]': finished 2023-07-12T15:39:54.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:54 smithi145 ceph-mon[95801]: osdmap e558: 8 total, 8 up, 8 in 2023-07-12T15:39:54.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:39:54.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:39:54.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-07-12T15:39:55.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:39:55 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:39:55.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:55 smithi069 ceph-mon[102777]: pgmap v843: 164 pgs: 164 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:55.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:39:55.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:55 smithi069 ceph-mon[102777]: osdmap e559: 8 total, 8 up, 8 in 2023-07-12T15:39:55.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:39:55.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:55 smithi069 ceph-mon[94241]: pgmap v843: 164 pgs: 164 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:55.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:39:55.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:55 smithi069 ceph-mon[94241]: osdmap e559: 8 total, 8 up, 8 in 2023-07-12T15:39:55.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:39:55.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:55 smithi145 ceph-mon[95801]: pgmap v843: 164 pgs: 164 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:39:55.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi069-133790-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-07-12T15:39:55.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:55 smithi145 ceph-mon[95801]: osdmap e559: 8 total, 8 up, 8 in 2023-07-12T15:39:55.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:39:56.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:39:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:39:56] "GET /metrics HTTP/1.1" 200 230881 "" "Prometheus/2.43.0" 2023-07-12T15:39:56.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:56 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:56.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:56 smithi069 ceph-mon[94241]: osdmap e560: 8 total, 8 up, 8 in 2023-07-12T15:39:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:56 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:56 smithi069 ceph-mon[102777]: osdmap e560: 8 total, 8 up, 8 in 2023-07-12T15:39:56.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:56 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:39:56.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:56 smithi145 ceph-mon[95801]: osdmap e560: 8 total, 8 up, 8 in 2023-07-12T15:39:57.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:57 smithi069 ceph-mon[102777]: pgmap v846: 164 pgs: 164 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:57.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:39:57.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:57 smithi069 ceph-mon[102777]: osdmap e561: 8 total, 8 up, 8 in 2023-07-12T15:39:57.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:57 smithi069 ceph-mon[102777]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:39:57.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:57 smithi069 ceph-mon[94241]: pgmap v846: 164 pgs: 164 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:57.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:39:57.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:57 smithi069 ceph-mon[94241]: osdmap e561: 8 total, 8 up, 8 in 2023-07-12T15:39:57.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:57 smithi069 ceph-mon[94241]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:39:57.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:57 smithi145 ceph-mon[95801]: pgmap v846: 164 pgs: 164 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:57.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi069-133790-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:39:57.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:57 smithi145 ceph-mon[95801]: osdmap e561: 8 total, 8 up, 8 in 2023-07-12T15:39:57.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:57 smithi145 ceph-mon[95801]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:39:58.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:58 smithi069 ceph-mon[102777]: osdmap e562: 8 total, 8 up, 8 in 2023-07-12T15:39:58.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:58.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:58 smithi069 ceph-mon[94241]: osdmap e562: 8 total, 8 up, 8 in 2023-07-12T15:39:58.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:58.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:58 smithi145 ceph-mon[95801]: osdmap e562: 8 total, 8 up, 8 in 2023-07-12T15:39:58.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:39:59.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:59 smithi069 ceph-mon[102777]: pgmap v849: 204 pgs: 40 unknown, 164 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:59.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-112","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:59.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:59 smithi069 ceph-mon[102777]: osdmap e563: 8 total, 8 up, 8 in 2023-07-12T15:39:59.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:59 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:39:59.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:39:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-112", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:39:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:59 smithi069 ceph-mon[94241]: pgmap v849: 204 pgs: 40 unknown, 164 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-112","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:59 smithi069 ceph-mon[94241]: osdmap e563: 8 total, 8 up, 8 in 2023-07-12T15:39:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:59 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:39:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:39:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-112", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:39:59.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:59 smithi145 ceph-mon[95801]: pgmap v849: 204 pgs: 40 unknown, 164 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:39:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-112","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:39:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:59 smithi145 ceph-mon[95801]: osdmap e563: 8 total, 8 up, 8 in 2023-07-12T15:39:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:59 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:39:59.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:39:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-112", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:00.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-112", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:00.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[102777]: osdmap e564: 8 total, 8 up, 8 in 2023-07-12T15:40:00.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-112"}]: dispatch 2023-07-12T15:40:00.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[102777]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2023-07-12T15:40:00.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[102777]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2023-07-12T15:40:00.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[102777]: application not enabled on pool 'ceph_test_rados_api_asio' 2023-07-12T15:40:00.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[102777]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2023-07-12T15:40:00.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-112", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:00.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[94241]: osdmap e564: 8 total, 8 up, 8 in 2023-07-12T15:40:00.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-112"}]: dispatch 2023-07-12T15:40:00.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[94241]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2023-07-12T15:40:00.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[94241]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2023-07-12T15:40:00.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[94241]: application not enabled on pool 'ceph_test_rados_api_asio' 2023-07-12T15:40:00.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:00 smithi069 ceph-mon[94241]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2023-07-12T15:40:00.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-112", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:00 smithi145 ceph-mon[95801]: osdmap e564: 8 total, 8 up, 8 in 2023-07-12T15:40:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-112"}]: dispatch 2023-07-12T15:40:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:00 smithi145 ceph-mon[95801]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2023-07-12T15:40:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:00 smithi145 ceph-mon[95801]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2023-07-12T15:40:00.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:00 smithi145 ceph-mon[95801]: application not enabled on pool 'ceph_test_rados_api_asio' 2023-07-12T15:40:00.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:00 smithi145 ceph-mon[95801]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2023-07-12T15:40:01.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:01 smithi069 ceph-mon[102777]: pgmap v852: 204 pgs: 204 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:40:01.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-112"}]': finished 2023-07-12T15:40:01.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:01 smithi069 ceph-mon[102777]: osdmap e565: 8 total, 8 up, 8 in 2023-07-12T15:40:01.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:01 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:01.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:01 smithi069 ceph-mon[94241]: pgmap v852: 204 pgs: 204 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:40:01.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-112"}]': finished 2023-07-12T15:40:01.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:01 smithi069 ceph-mon[94241]: osdmap e565: 8 total, 8 up, 8 in 2023-07-12T15:40:01.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:01 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:01.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:01 smithi145 ceph-mon[95801]: pgmap v852: 204 pgs: 204 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:40:01.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-112"}]': finished 2023-07-12T15:40:01.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:01 smithi145 ceph-mon[95801]: osdmap e565: 8 total, 8 up, 8 in 2023-07-12T15:40:01.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:01 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:02.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:02.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:02 smithi069 ceph-mon[102777]: osdmap e566: 8 total, 8 up, 8 in 2023-07-12T15:40:02.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-112"}]: dispatch 2023-07-12T15:40:02.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:02 smithi069 ceph-mon[102777]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:40:02.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-112"}]': finished 2023-07-12T15:40:02.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:02 smithi069 ceph-mon[102777]: osdmap e567: 8 total, 8 up, 8 in 2023-07-12T15:40:02.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:02.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:02 smithi069 ceph-mon[94241]: osdmap e566: 8 total, 8 up, 8 in 2023-07-12T15:40:02.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-112"}]: dispatch 2023-07-12T15:40:02.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:02 smithi069 ceph-mon[94241]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:40:02.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-112"}]': finished 2023-07-12T15:40:02.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:02 smithi069 ceph-mon[94241]: osdmap e567: 8 total, 8 up, 8 in 2023-07-12T15:40:02.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:02.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:02 smithi145 ceph-mon[95801]: osdmap e566: 8 total, 8 up, 8 in 2023-07-12T15:40:02.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-112"}]: dispatch 2023-07-12T15:40:02.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:02 smithi145 ceph-mon[95801]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:40:02.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-112"}]': finished 2023-07-12T15:40:02.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:02 smithi145 ceph-mon[95801]: osdmap e567: 8 total, 8 up, 8 in 2023-07-12T15:40:03.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:03 smithi069 ceph-mon[102777]: pgmap v855: 204 pgs: 204 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:40:03.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:03 smithi069 ceph-mon[94241]: pgmap v855: 204 pgs: 204 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:40:03.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:03 smithi145 ceph-mon[95801]: pgmap v855: 204 pgs: 204 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:40:04.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:04 smithi069 ceph-mon[102777]: osdmap e568: 8 total, 8 up, 8 in 2023-07-12T15:40:04.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:04 smithi069 ceph-mon[94241]: osdmap e568: 8 total, 8 up, 8 in 2023-07-12T15:40:04.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:04 smithi145 ceph-mon[95801]: osdmap e568: 8 total, 8 up, 8 in 2023-07-12T15:40:05.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:40:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:40:05.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:05 smithi069 ceph-mon[102777]: pgmap v858: 172 pgs: 172 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:40:05.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:05 smithi069 ceph-mon[102777]: osdmap e569: 8 total, 8 up, 8 in 2023-07-12T15:40:05.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:05.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:05 smithi069 ceph-mon[94241]: pgmap v858: 172 pgs: 172 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:40:05.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:05 smithi069 ceph-mon[94241]: osdmap e569: 8 total, 8 up, 8 in 2023-07-12T15:40:05.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:05.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:05 smithi145 ceph-mon[95801]: pgmap v858: 172 pgs: 172 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:40:05.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:05 smithi145 ceph-mon[95801]: osdmap e569: 8 total, 8 up, 8 in 2023-07-12T15:40:05.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:06.442 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:40:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:40:06] "GET /metrics HTTP/1.1" 200 228457 "" "Prometheus/2.43.0" 2023-07-12T15:40:06.443 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:06 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:06.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-114","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:06.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:06 smithi069 ceph-mon[94241]: osdmap e570: 8 total, 8 up, 8 in 2023-07-12T15:40:06.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-114", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:06.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:06 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:06.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-114","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:06.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:06 smithi069 ceph-mon[102777]: osdmap e570: 8 total, 8 up, 8 in 2023-07-12T15:40:06.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-114", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:06 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-114","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:06 smithi145 ceph-mon[95801]: osdmap e570: 8 total, 8 up, 8 in 2023-07-12T15:40:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-114", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:07.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:07 smithi069 ceph-mon[102777]: pgmap v861: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-12T15:40:07.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-114", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:07.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:07 smithi069 ceph-mon[102777]: osdmap e571: 8 total, 8 up, 8 in 2023-07-12T15:40:07.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-114"}]: dispatch 2023-07-12T15:40:07.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:07 smithi069 ceph-mon[94241]: pgmap v861: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-12T15:40:07.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-114", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:07.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:07 smithi069 ceph-mon[94241]: osdmap e571: 8 total, 8 up, 8 in 2023-07-12T15:40:07.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-114"}]: dispatch 2023-07-12T15:40:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:07 smithi145 ceph-mon[95801]: pgmap v861: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-12T15:40:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-114", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:07 smithi145 ceph-mon[95801]: osdmap e571: 8 total, 8 up, 8 in 2023-07-12T15:40:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-114"}]: dispatch 2023-07-12T15:40:08.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-114"}]': finished 2023-07-12T15:40:08.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:08 smithi069 ceph-mon[94241]: osdmap e572: 8 total, 8 up, 8 in 2023-07-12T15:40:08.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-114", "mode": "writeback"}]: dispatch 2023-07-12T15:40:08.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:08 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:08.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-114", "mode": "writeback"}]': finished 2023-07-12T15:40:08.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:08 smithi069 ceph-mon[94241]: osdmap e573: 8 total, 8 up, 8 in 2023-07-12T15:40:08.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-114"}]': finished 2023-07-12T15:40:08.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:08 smithi069 ceph-mon[102777]: osdmap e572: 8 total, 8 up, 8 in 2023-07-12T15:40:08.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-114", "mode": "writeback"}]: dispatch 2023-07-12T15:40:08.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:08 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:08.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-114", "mode": "writeback"}]': finished 2023-07-12T15:40:08.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:08 smithi069 ceph-mon[102777]: osdmap e573: 8 total, 8 up, 8 in 2023-07-12T15:40:08.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-114"}]': finished 2023-07-12T15:40:08.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:08 smithi145 ceph-mon[95801]: osdmap e572: 8 total, 8 up, 8 in 2023-07-12T15:40:08.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-114", "mode": "writeback"}]: dispatch 2023-07-12T15:40:08.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:08 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:08.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-114", "mode": "writeback"}]': finished 2023-07-12T15:40:08.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:08 smithi145 ceph-mon[95801]: osdmap e573: 8 total, 8 up, 8 in 2023-07-12T15:40:09.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:09 smithi069 ceph-mon[102777]: pgmap v864: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-12T15:40:09.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:09.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:09 smithi069 ceph-mon[94241]: pgmap v864: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-12T15:40:09.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:09.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:09 smithi145 ceph-mon[95801]: pgmap v864: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2023-07-12T15:40:09.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:10.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:10.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:10 smithi069 ceph-mon[102777]: osdmap e574: 8 total, 8 up, 8 in 2023-07-12T15:40:10.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-114"}]: dispatch 2023-07-12T15:40:10.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:10.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:10 smithi069 ceph-mon[94241]: osdmap e574: 8 total, 8 up, 8 in 2023-07-12T15:40:10.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-114"}]: dispatch 2023-07-12T15:40:10.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:10.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:10 smithi145 ceph-mon[95801]: osdmap e574: 8 total, 8 up, 8 in 2023-07-12T15:40:10.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-114"}]: dispatch 2023-07-12T15:40:11.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:11 smithi069 ceph-mon[94241]: pgmap v867: 204 pgs: 204 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:40:11.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:11 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:40:11.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-114"}]': finished 2023-07-12T15:40:11.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:11 smithi069 ceph-mon[94241]: osdmap e575: 8 total, 8 up, 8 in 2023-07-12T15:40:11.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:11 smithi069 ceph-mon[102777]: pgmap v867: 204 pgs: 204 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:40:11.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:11 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:40:11.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-114"}]': finished 2023-07-12T15:40:11.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:11 smithi069 ceph-mon[102777]: osdmap e575: 8 total, 8 up, 8 in 2023-07-12T15:40:12.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:11 smithi145 ceph-mon[95801]: pgmap v867: 204 pgs: 204 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2023-07-12T15:40:12.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:11 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:40:12.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-114"}]': finished 2023-07-12T15:40:12.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:11 smithi145 ceph-mon[95801]: osdmap e575: 8 total, 8 up, 8 in 2023-07-12T15:40:12.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:12 smithi069 ceph-mon[102777]: osdmap e576: 8 total, 8 up, 8 in 2023-07-12T15:40:12.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:12 smithi069 ceph-mon[102777]: osdmap e577: 8 total, 8 up, 8 in 2023-07-12T15:40:12.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:12.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:12 smithi069 ceph-mon[94241]: osdmap e576: 8 total, 8 up, 8 in 2023-07-12T15:40:12.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:12 smithi069 ceph-mon[94241]: osdmap e577: 8 total, 8 up, 8 in 2023-07-12T15:40:12.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:13.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:12 smithi145 ceph-mon[95801]: osdmap e576: 8 total, 8 up, 8 in 2023-07-12T15:40:13.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:12 smithi145 ceph-mon[95801]: osdmap e577: 8 total, 8 up, 8 in 2023-07-12T15:40:13.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:13.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:13 smithi069 ceph-mon[94241]: pgmap v870: 172 pgs: 172 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:13.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:40:13.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-116","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:13.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:13 smithi069 ceph-mon[94241]: osdmap e578: 8 total, 8 up, 8 in 2023-07-12T15:40:13.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:13 smithi069 ceph-mon[102777]: pgmap v870: 172 pgs: 172 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:13.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:40:13.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-116","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:13.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:13 smithi069 ceph-mon[102777]: osdmap e578: 8 total, 8 up, 8 in 2023-07-12T15:40:14.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:13 smithi145 ceph-mon[95801]: pgmap v870: 172 pgs: 172 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:14.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:40:14.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-116","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:14.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:13 smithi145 ceph-mon[95801]: osdmap e578: 8 total, 8 up, 8 in 2023-07-12T15:40:15.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:40:15 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:40:15.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:15 smithi069 ceph-mon[102777]: pgmap v873: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:40:15.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:15 smithi069 ceph-mon[102777]: osdmap e579: 8 total, 8 up, 8 in 2023-07-12T15:40:15.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-116", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:15.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:15 smithi069 ceph-mon[94241]: pgmap v873: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:40:15.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:15 smithi069 ceph-mon[94241]: osdmap e579: 8 total, 8 up, 8 in 2023-07-12T15:40:15.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-116", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:15.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:15 smithi145 ceph-mon[95801]: pgmap v873: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:40:15.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:15 smithi145 ceph-mon[95801]: osdmap e579: 8 total, 8 up, 8 in 2023-07-12T15:40:15.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-116", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:16.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:40:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:40:16] "GET /metrics HTTP/1.1" 200 228508 "" "Prometheus/2.43.0" 2023-07-12T15:40:16.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:16 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:16.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-116", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:16.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:16 smithi145 ceph-mon[95801]: osdmap e580: 8 total, 8 up, 8 in 2023-07-12T15:40:16.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-116"}]: dispatch 2023-07-12T15:40:16.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:16 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:16.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-116", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:16.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:16 smithi069 ceph-mon[94241]: osdmap e580: 8 total, 8 up, 8 in 2023-07-12T15:40:16.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-116"}]: dispatch 2023-07-12T15:40:16.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:16 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:16.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-116", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:16.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:16 smithi069 ceph-mon[102777]: osdmap e580: 8 total, 8 up, 8 in 2023-07-12T15:40:16.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-116"}]: dispatch 2023-07-12T15:40:17.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:17 smithi145 ceph-mon[95801]: pgmap v876: 204 pgs: 204 active+clean; 582 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-12T15:40:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-116"}]': finished 2023-07-12T15:40:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:17 smithi145 ceph-mon[95801]: osdmap e581: 8 total, 8 up, 8 in 2023-07-12T15:40:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-116", "mode": "writeback"}]: dispatch 2023-07-12T15:40:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:17 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-116", "mode": "writeback"}]': finished 2023-07-12T15:40:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:17 smithi145 ceph-mon[95801]: osdmap e582: 8 total, 8 up, 8 in 2023-07-12T15:40:17.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[94241]: pgmap v876: 204 pgs: 204 active+clean; 582 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-12T15:40:17.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-116"}]': finished 2023-07-12T15:40:17.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[94241]: osdmap e581: 8 total, 8 up, 8 in 2023-07-12T15:40:17.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-116", "mode": "writeback"}]: dispatch 2023-07-12T15:40:17.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:17.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-116", "mode": "writeback"}]': finished 2023-07-12T15:40:17.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[94241]: osdmap e582: 8 total, 8 up, 8 in 2023-07-12T15:40:17.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[102777]: pgmap v876: 204 pgs: 204 active+clean; 582 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-12T15:40:17.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-116"}]': finished 2023-07-12T15:40:17.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[102777]: osdmap e581: 8 total, 8 up, 8 in 2023-07-12T15:40:17.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-116", "mode": "writeback"}]: dispatch 2023-07-12T15:40:17.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:17.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-116", "mode": "writeback"}]': finished 2023-07-12T15:40:17.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:17 smithi069 ceph-mon[102777]: osdmap e582: 8 total, 8 up, 8 in 2023-07-12T15:40:18.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "244.6"}]: dispatch 2023-07-12T15:40:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:18 smithi145 ceph-mon[95801]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "244.6"}]: dispatch 2023-07-12T15:40:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:18 smithi145 ceph-mon[95801]: 244.6 scrub starts 2023-07-12T15:40:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:18 smithi145 ceph-mon[95801]: 244.6 scrub ok 2023-07-12T15:40:18.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "244.6"}]: dispatch 2023-07-12T15:40:18.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:18 smithi069 ceph-mon[94241]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "244.6"}]: dispatch 2023-07-12T15:40:18.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:18 smithi069 ceph-mon[94241]: 244.6 scrub starts 2023-07-12T15:40:18.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:18 smithi069 ceph-mon[94241]: 244.6 scrub ok 2023-07-12T15:40:18.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "244.6"}]: dispatch 2023-07-12T15:40:18.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:18 smithi069 ceph-mon[102777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "244.6"}]: dispatch 2023-07-12T15:40:18.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:18 smithi069 ceph-mon[102777]: 244.6 scrub starts 2023-07-12T15:40:18.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:18 smithi069 ceph-mon[102777]: 244.6 scrub ok 2023-07-12T15:40:19.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:19 smithi145 ceph-mon[95801]: pgmap v879: 204 pgs: 204 active+clean; 582 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-12T15:40:19.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:19 smithi069 ceph-mon[94241]: pgmap v879: 204 pgs: 204 active+clean; 582 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-12T15:40:19.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:19 smithi069 ceph-mon[102777]: pgmap v879: 204 pgs: 204 active+clean; 582 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2023-07-12T15:40:21.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:21 smithi145 ceph-mon[95801]: pgmap v880: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2023-07-12T15:40:21.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:21 smithi069 ceph-mon[94241]: pgmap v880: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2023-07-12T15:40:21.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:21 smithi069 ceph-mon[102777]: pgmap v880: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2023-07-12T15:40:23.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:22 smithi069 ceph-mon[102777]: pgmap v881: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 152 B/s wr, 1 op/s 2023-07-12T15:40:23.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:22 smithi069 ceph-mon[94241]: pgmap v881: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 152 B/s wr, 1 op/s 2023-07-12T15:40:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:22 smithi145 ceph-mon[95801]: pgmap v881: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 152 B/s wr, 1 op/s 2023-07-12T15:40:25.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:40:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:40:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:25 smithi145 ceph-mon[95801]: pgmap v882: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 127 B/s wr, 1 op/s 2023-07-12T15:40:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:40:25.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:25 smithi069 ceph-mon[102777]: pgmap v882: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 127 B/s wr, 1 op/s 2023-07-12T15:40:25.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:40:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:25 smithi069 ceph-mon[94241]: pgmap v882: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 127 B/s wr, 1 op/s 2023-07-12T15:40:25.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:40:26.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:40:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:40:26] "GET /metrics HTTP/1.1" 200 228508 "" "Prometheus/2.43.0" 2023-07-12T15:40:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:26 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:26.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:26 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:40:26.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:26 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:40:26.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:26 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:40:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:26 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:26 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:40:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:26 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:40:26.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:26 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:40:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:26 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:26 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:40:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:26 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:40:26.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:26 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:40:27.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:27 smithi069 ceph-mon[102777]: pgmap v883: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 B/s wr, 1 op/s 2023-07-12T15:40:27.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:27 smithi069 ceph-mon[94241]: pgmap v883: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 B/s wr, 1 op/s 2023-07-12T15:40:27.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:27 smithi145 ceph-mon[95801]: pgmap v883: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 B/s wr, 1 op/s 2023-07-12T15:40:29.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:29 smithi069 ceph-mon[94241]: pgmap v884: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 B/s wr, 1 op/s 2023-07-12T15:40:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:29 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:40:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:40:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:29 smithi069 ceph-mon[102777]: pgmap v884: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 B/s wr, 1 op/s 2023-07-12T15:40:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:29 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:40:29.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:40:29.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:29 smithi145 ceph-mon[95801]: pgmap v884: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 B/s wr, 1 op/s 2023-07-12T15:40:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:29 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:40:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:40:31.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:31 smithi069 ceph-mon[94241]: pgmap v885: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 85 B/s wr, 1 op/s 2023-07-12T15:40:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:31 smithi069 ceph-mon[102777]: pgmap v885: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 85 B/s wr, 1 op/s 2023-07-12T15:40:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:31 smithi145 ceph-mon[95801]: pgmap v885: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 85 B/s wr, 1 op/s 2023-07-12T15:40:33.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:32 smithi069 ceph-mon[102777]: pgmap v886: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:40:33.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:32 smithi069 ceph-mon[94241]: pgmap v886: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:40:33.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:32 smithi145 ceph-mon[95801]: pgmap v886: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:40:34.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:33 smithi069 ceph-mon[102777]: osdmap e583: 8 total, 8 up, 8 in 2023-07-12T15:40:34.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:34.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:33 smithi069 ceph-mon[94241]: osdmap e583: 8 total, 8 up, 8 in 2023-07-12T15:40:34.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:34.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:33 smithi145 ceph-mon[95801]: osdmap e583: 8 total, 8 up, 8 in 2023-07-12T15:40:34.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:35.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[102777]: osdmap e584: 8 total, 8 up, 8 in 2023-07-12T15:40:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-116"}]: dispatch 2023-07-12T15:40:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[102777]: pgmap v889: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:40:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:40:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-116"}]': finished 2023-07-12T15:40:35.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[102777]: osdmap e585: 8 total, 8 up, 8 in 2023-07-12T15:40:35.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:35.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[94241]: osdmap e584: 8 total, 8 up, 8 in 2023-07-12T15:40:35.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-116"}]: dispatch 2023-07-12T15:40:35.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[94241]: pgmap v889: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:40:35.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:40:35.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-116"}]': finished 2023-07-12T15:40:35.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:34 smithi069 ceph-mon[94241]: osdmap e585: 8 total, 8 up, 8 in 2023-07-12T15:40:35.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:40:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:40:35.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:35.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:34 smithi145 ceph-mon[95801]: osdmap e584: 8 total, 8 up, 8 in 2023-07-12T15:40:35.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-116"}]: dispatch 2023-07-12T15:40:35.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:34 smithi145 ceph-mon[95801]: pgmap v889: 204 pgs: 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:40:35.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:34 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:40:35.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-116"}]': finished 2023-07-12T15:40:35.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:34 smithi145 ceph-mon[95801]: osdmap e585: 8 total, 8 up, 8 in 2023-07-12T15:40:36.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:36 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:36.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:36 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:36.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:40:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:40:36] "GET /metrics HTTP/1.1" 200 228504 "" "Prometheus/2.43.0" 2023-07-12T15:40:36.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:36 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:37.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:37 smithi069 ceph-mon[94241]: osdmap e586: 8 total, 8 up, 8 in 2023-07-12T15:40:37.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:37 smithi069 ceph-mon[94241]: pgmap v892: 172 pgs: 172 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2023-07-12T15:40:37.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:37 smithi069 ceph-mon[94241]: osdmap e587: 8 total, 8 up, 8 in 2023-07-12T15:40:37.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:37.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:37 smithi069 ceph-mon[102777]: osdmap e586: 8 total, 8 up, 8 in 2023-07-12T15:40:37.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:37 smithi069 ceph-mon[102777]: pgmap v892: 172 pgs: 172 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2023-07-12T15:40:37.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:37 smithi069 ceph-mon[102777]: osdmap e587: 8 total, 8 up, 8 in 2023-07-12T15:40:37.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:37.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:37 smithi145 ceph-mon[95801]: osdmap e586: 8 total, 8 up, 8 in 2023-07-12T15:40:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:37 smithi145 ceph-mon[95801]: pgmap v892: 172 pgs: 172 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2023-07-12T15:40:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:37 smithi145 ceph-mon[95801]: osdmap e587: 8 total, 8 up, 8 in 2023-07-12T15:40:37.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:39.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-118","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:39.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:38 smithi069 ceph-mon[94241]: osdmap e588: 8 total, 8 up, 8 in 2023-07-12T15:40:39.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:38 smithi069 ceph-mon[94241]: pgmap v895: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2023-07-12T15:40:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-118","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:39.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:38 smithi069 ceph-mon[102777]: osdmap e588: 8 total, 8 up, 8 in 2023-07-12T15:40:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:38 smithi069 ceph-mon[102777]: pgmap v895: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2023-07-12T15:40:39.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-118","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:39.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:38 smithi145 ceph-mon[95801]: osdmap e588: 8 total, 8 up, 8 in 2023-07-12T15:40:39.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:38 smithi145 ceph-mon[95801]: pgmap v895: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2023-07-12T15:40:40.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:39 smithi069 ceph-mon[102777]: osdmap e589: 8 total, 8 up, 8 in 2023-07-12T15:40:40.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-118", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:40.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:39 smithi069 ceph-mon[94241]: osdmap e589: 8 total, 8 up, 8 in 2023-07-12T15:40:40.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-118", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:39 smithi145 ceph-mon[95801]: osdmap e589: 8 total, 8 up, 8 in 2023-07-12T15:40:40.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-118", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:41.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-118", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:41.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:40 smithi069 ceph-mon[102777]: osdmap e590: 8 total, 8 up, 8 in 2023-07-12T15:40:41.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-118"}]: dispatch 2023-07-12T15:40:41.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:40 smithi069 ceph-mon[102777]: pgmap v898: 204 pgs: 204 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:40:41.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-118"}]': finished 2023-07-12T15:40:41.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:40 smithi069 ceph-mon[102777]: osdmap e591: 8 total, 8 up, 8 in 2023-07-12T15:40:41.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-118", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:41.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:40 smithi069 ceph-mon[94241]: osdmap e590: 8 total, 8 up, 8 in 2023-07-12T15:40:41.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-118"}]: dispatch 2023-07-12T15:40:41.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:40 smithi069 ceph-mon[94241]: pgmap v898: 204 pgs: 204 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:40:41.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-118"}]': finished 2023-07-12T15:40:41.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:40 smithi069 ceph-mon[94241]: osdmap e591: 8 total, 8 up, 8 in 2023-07-12T15:40:41.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-118", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:40 smithi145 ceph-mon[95801]: osdmap e590: 8 total, 8 up, 8 in 2023-07-12T15:40:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-118"}]: dispatch 2023-07-12T15:40:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:40 smithi145 ceph-mon[95801]: pgmap v898: 204 pgs: 204 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:40:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-118"}]': finished 2023-07-12T15:40:41.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:40 smithi145 ceph-mon[95801]: osdmap e591: 8 total, 8 up, 8 in 2023-07-12T15:40:42.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-118", "mode": "writeback"}]: dispatch 2023-07-12T15:40:42.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:41 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:42.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-118", "mode": "writeback"}]': finished 2023-07-12T15:40:42.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:41 smithi069 ceph-mon[102777]: osdmap e592: 8 total, 8 up, 8 in 2023-07-12T15:40:42.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-118", "mode": "writeback"}]: dispatch 2023-07-12T15:40:42.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:41 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:42.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-118", "mode": "writeback"}]': finished 2023-07-12T15:40:42.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:41 smithi069 ceph-mon[94241]: osdmap e592: 8 total, 8 up, 8 in 2023-07-12T15:40:42.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-118", "mode": "writeback"}]: dispatch 2023-07-12T15:40:42.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:41 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:42.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-118", "mode": "writeback"}]': finished 2023-07-12T15:40:42.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:41 smithi145 ceph-mon[95801]: osdmap e592: 8 total, 8 up, 8 in 2023-07-12T15:40:43.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:42 smithi069 ceph-mon[94241]: pgmap v901: 204 pgs: 204 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:40:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:42 smithi069 ceph-mon[94241]: osdmap e593: 8 total, 8 up, 8 in 2023-07-12T15:40:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:42 smithi069 ceph-mon[102777]: pgmap v901: 204 pgs: 204 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:40:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:42 smithi069 ceph-mon[102777]: osdmap e593: 8 total, 8 up, 8 in 2023-07-12T15:40:43.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:42 smithi145 ceph-mon[95801]: pgmap v901: 204 pgs: 204 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:40:43.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:42 smithi145 ceph-mon[95801]: osdmap e593: 8 total, 8 up, 8 in 2023-07-12T15:40:44.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:43 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:40:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:43 smithi069 ceph-mon[102777]: osdmap e594: 8 total, 8 up, 8 in 2023-07-12T15:40:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-118"}]: dispatch 2023-07-12T15:40:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:43 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:40:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:43 smithi069 ceph-mon[94241]: osdmap e594: 8 total, 8 up, 8 in 2023-07-12T15:40:44.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-118"}]: dispatch 2023-07-12T15:40:44.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:44.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:43 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:40:44.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:44.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:43 smithi145 ceph-mon[95801]: osdmap e594: 8 total, 8 up, 8 in 2023-07-12T15:40:44.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-118"}]: dispatch 2023-07-12T15:40:45.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:44 smithi069 ceph-mon[102777]: pgmap v904: 204 pgs: 204 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:40:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:44 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:40:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-118"}]': finished 2023-07-12T15:40:45.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:44 smithi069 ceph-mon[102777]: osdmap e595: 8 total, 8 up, 8 in 2023-07-12T15:40:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:44 smithi069 ceph-mon[94241]: pgmap v904: 204 pgs: 204 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:40:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:44 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:40:45.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-118"}]': finished 2023-07-12T15:40:45.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:44 smithi069 ceph-mon[94241]: osdmap e595: 8 total, 8 up, 8 in 2023-07-12T15:40:45.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:40:45 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:40:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:44 smithi145 ceph-mon[95801]: pgmap v904: 204 pgs: 204 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:40:45.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:44 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:40:45.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-118"}]': finished 2023-07-12T15:40:45.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:44 smithi145 ceph-mon[95801]: osdmap e595: 8 total, 8 up, 8 in 2023-07-12T15:40:46.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:46 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:46 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:46.487 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:40:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:40:46] "GET /metrics HTTP/1.1" 200 228500 "" "Prometheus/2.43.0" 2023-07-12T15:40:46.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:46 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:47.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:47 smithi069 ceph-mon[102777]: osdmap e596: 8 total, 8 up, 8 in 2023-07-12T15:40:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:47 smithi069 ceph-mon[102777]: pgmap v907: 172 pgs: 172 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:40:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:47 smithi069 ceph-mon[102777]: osdmap e597: 8 total, 8 up, 8 in 2023-07-12T15:40:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:47 smithi069 ceph-mon[94241]: osdmap e596: 8 total, 8 up, 8 in 2023-07-12T15:40:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:47 smithi069 ceph-mon[94241]: pgmap v907: 172 pgs: 172 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:40:47.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:47 smithi069 ceph-mon[94241]: osdmap e597: 8 total, 8 up, 8 in 2023-07-12T15:40:47.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:47.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:47 smithi145 ceph-mon[95801]: osdmap e596: 8 total, 8 up, 8 in 2023-07-12T15:40:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:47 smithi145 ceph-mon[95801]: pgmap v907: 172 pgs: 172 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:40:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:47 smithi145 ceph-mon[95801]: osdmap e597: 8 total, 8 up, 8 in 2023-07-12T15:40:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:49.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-120","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:49.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:48 smithi069 ceph-mon[102777]: osdmap e598: 8 total, 8 up, 8 in 2023-07-12T15:40:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-120", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:49.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:48 smithi069 ceph-mon[102777]: pgmap v910: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:40:49.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-120","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:49.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:48 smithi069 ceph-mon[94241]: osdmap e598: 8 total, 8 up, 8 in 2023-07-12T15:40:49.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-120", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:49.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:48 smithi069 ceph-mon[94241]: pgmap v910: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:40:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-120","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:48 smithi145 ceph-mon[95801]: osdmap e598: 8 total, 8 up, 8 in 2023-07-12T15:40:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-120", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:48 smithi145 ceph-mon[95801]: pgmap v910: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:40:50.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-120", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:50.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:49 smithi069 ceph-mon[102777]: osdmap e599: 8 total, 8 up, 8 in 2023-07-12T15:40:50.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-120"}]: dispatch 2023-07-12T15:40:50.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-120", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:50.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:49 smithi069 ceph-mon[94241]: osdmap e599: 8 total, 8 up, 8 in 2023-07-12T15:40:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-120"}]: dispatch 2023-07-12T15:40:50.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-120", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:50.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:49 smithi145 ceph-mon[95801]: osdmap e599: 8 total, 8 up, 8 in 2023-07-12T15:40:50.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-120"}]: dispatch 2023-07-12T15:40:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-120"}]': finished 2023-07-12T15:40:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[102777]: osdmap e600: 8 total, 8 up, 8 in 2023-07-12T15:40:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-120", "mode": "writeback"}]: dispatch 2023-07-12T15:40:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[102777]: pgmap v913: 204 pgs: 204 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-120", "mode": "writeback"}]': finished 2023-07-12T15:40:51.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[102777]: osdmap e601: 8 total, 8 up, 8 in 2023-07-12T15:40:51.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-120"}]': finished 2023-07-12T15:40:51.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[94241]: osdmap e600: 8 total, 8 up, 8 in 2023-07-12T15:40:51.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-120", "mode": "writeback"}]: dispatch 2023-07-12T15:40:51.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[94241]: pgmap v913: 204 pgs: 204 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:51.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:51.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-120", "mode": "writeback"}]': finished 2023-07-12T15:40:51.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:50 smithi069 ceph-mon[94241]: osdmap e601: 8 total, 8 up, 8 in 2023-07-12T15:40:51.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-120"}]': finished 2023-07-12T15:40:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:50 smithi145 ceph-mon[95801]: osdmap e600: 8 total, 8 up, 8 in 2023-07-12T15:40:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-120", "mode": "writeback"}]: dispatch 2023-07-12T15:40:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:50 smithi145 ceph-mon[95801]: pgmap v913: 204 pgs: 204 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:50 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-120", "mode": "writeback"}]': finished 2023-07-12T15:40:51.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:50 smithi145 ceph-mon[95801]: osdmap e601: 8 total, 8 up, 8 in 2023-07-12T15:40:52.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:51 smithi145 ceph-mon[95801]: osdmap e602: 8 total, 8 up, 8 in 2023-07-12T15:40:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-120"}]: dispatch 2023-07-12T15:40:52.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:52.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:52.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:51 smithi069 ceph-mon[94241]: osdmap e602: 8 total, 8 up, 8 in 2023-07-12T15:40:52.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-120"}]: dispatch 2023-07-12T15:40:52.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:40:52.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:40:52.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:51 smithi069 ceph-mon[102777]: osdmap e602: 8 total, 8 up, 8 in 2023-07-12T15:40:52.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-120"}]: dispatch 2023-07-12T15:40:53.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:52 smithi145 ceph-mon[95801]: pgmap v916: 204 pgs: 204 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:52 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:40:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-120"}]': finished 2023-07-12T15:40:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:52 smithi145 ceph-mon[95801]: osdmap e603: 8 total, 8 up, 8 in 2023-07-12T15:40:53.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:52 smithi069 ceph-mon[102777]: pgmap v916: 204 pgs: 204 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:53.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:52 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:40:53.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-120"}]': finished 2023-07-12T15:40:53.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:52 smithi069 ceph-mon[102777]: osdmap e603: 8 total, 8 up, 8 in 2023-07-12T15:40:53.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:52 smithi069 ceph-mon[94241]: pgmap v916: 204 pgs: 204 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:53.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:52 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:40:53.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-120"}]': finished 2023-07-12T15:40:53.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:52 smithi069 ceph-mon[94241]: osdmap e603: 8 total, 8 up, 8 in 2023-07-12T15:40:55.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:54 smithi069 ceph-mon[102777]: osdmap e604: 8 total, 8 up, 8 in 2023-07-12T15:40:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:54 smithi069 ceph-mon[102777]: pgmap v919: 172 pgs: 172 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:40:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:54 smithi069 ceph-mon[94241]: osdmap e604: 8 total, 8 up, 8 in 2023-07-12T15:40:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:54 smithi069 ceph-mon[94241]: pgmap v919: 172 pgs: 172 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:40:55.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:40:55 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:40:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:54 smithi145 ceph-mon[95801]: osdmap e604: 8 total, 8 up, 8 in 2023-07-12T15:40:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:54 smithi145 ceph-mon[95801]: pgmap v919: 172 pgs: 172 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:40:56.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:40:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:40:56] "GET /metrics HTTP/1.1" 200 228500 "" "Prometheus/2.43.0" 2023-07-12T15:40:56.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:55 smithi069 ceph-mon[94241]: osdmap e605: 8 total, 8 up, 8 in 2023-07-12T15:40:56.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:56.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:55 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:56.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:55 smithi069 ceph-mon[102777]: osdmap e605: 8 total, 8 up, 8 in 2023-07-12T15:40:56.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:56.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:55 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:55 smithi145 ceph-mon[95801]: osdmap e605: 8 total, 8 up, 8 in 2023-07-12T15:40:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:40:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:55 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:40:57.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-122","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:56 smithi069 ceph-mon[102777]: osdmap e606: 8 total, 8 up, 8 in 2023-07-12T15:40:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-122", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:56 smithi069 ceph-mon[102777]: pgmap v922: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:57.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-122","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:57.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:56 smithi069 ceph-mon[94241]: osdmap e606: 8 total, 8 up, 8 in 2023-07-12T15:40:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-122", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:56 smithi069 ceph-mon[94241]: pgmap v922: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:57.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-122","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:40:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:56 smithi145 ceph-mon[95801]: osdmap e606: 8 total, 8 up, 8 in 2023-07-12T15:40:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-122", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:40:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:56 smithi145 ceph-mon[95801]: pgmap v922: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:58.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-122", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:58.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:57 smithi069 ceph-mon[102777]: osdmap e607: 8 total, 8 up, 8 in 2023-07-12T15:40:58.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-122"}]: dispatch 2023-07-12T15:40:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-122", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:58.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:57 smithi069 ceph-mon[94241]: osdmap e607: 8 total, 8 up, 8 in 2023-07-12T15:40:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-122"}]: dispatch 2023-07-12T15:40:58.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-122", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:40:58.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:57 smithi145 ceph-mon[95801]: osdmap e607: 8 total, 8 up, 8 in 2023-07-12T15:40:58.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-122"}]: dispatch 2023-07-12T15:40:59.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-122"}]': finished 2023-07-12T15:40:59.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[102777]: osdmap e608: 8 total, 8 up, 8 in 2023-07-12T15:40:59.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-122", "mode": "writeback"}]: dispatch 2023-07-12T15:40:59.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[102777]: pgmap v925: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:59.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:40:59.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:40:59.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:59.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-122"}]': finished 2023-07-12T15:40:59.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[94241]: osdmap e608: 8 total, 8 up, 8 in 2023-07-12T15:40:59.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-122", "mode": "writeback"}]: dispatch 2023-07-12T15:40:59.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[94241]: pgmap v925: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:59.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:40:59.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:40:59.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:58 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:40:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-122"}]': finished 2023-07-12T15:40:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:58 smithi145 ceph-mon[95801]: osdmap e608: 8 total, 8 up, 8 in 2023-07-12T15:40:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-122", "mode": "writeback"}]: dispatch 2023-07-12T15:40:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:58 smithi145 ceph-mon[95801]: pgmap v925: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:40:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:58 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:40:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:58 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:40:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:58 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:00.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-122", "mode": "writeback"}]': finished 2023-07-12T15:41:00.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:59 smithi069 ceph-mon[94241]: osdmap e609: 8 total, 8 up, 8 in 2023-07-12T15:41:00.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:40:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:00.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-122", "mode": "writeback"}]': finished 2023-07-12T15:41:00.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:59 smithi069 ceph-mon[102777]: osdmap e609: 8 total, 8 up, 8 in 2023-07-12T15:41:00.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:40:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:00.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-122", "mode": "writeback"}]': finished 2023-07-12T15:41:00.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:59 smithi145 ceph-mon[95801]: osdmap e609: 8 total, 8 up, 8 in 2023-07-12T15:41:00.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:40:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:00 smithi069 ceph-mon[102777]: osdmap e610: 8 total, 8 up, 8 in 2023-07-12T15:41:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-122"}]: dispatch 2023-07-12T15:41:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:00 smithi069 ceph-mon[102777]: pgmap v928: 204 pgs: 204 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2023-07-12T15:41:01.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:01.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:00 smithi069 ceph-mon[94241]: osdmap e610: 8 total, 8 up, 8 in 2023-07-12T15:41:01.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-122"}]: dispatch 2023-07-12T15:41:01.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:00 smithi069 ceph-mon[94241]: pgmap v928: 204 pgs: 204 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2023-07-12T15:41:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:00 smithi145 ceph-mon[95801]: osdmap e610: 8 total, 8 up, 8 in 2023-07-12T15:41:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-122"}]: dispatch 2023-07-12T15:41:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:00 smithi145 ceph-mon[95801]: pgmap v928: 204 pgs: 204 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2023-07-12T15:41:02.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:02 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-122"}]': finished 2023-07-12T15:41:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:02 smithi069 ceph-mon[102777]: osdmap e611: 8 total, 8 up, 8 in 2023-07-12T15:41:02.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:02 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:02.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-122"}]': finished 2023-07-12T15:41:02.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:02 smithi069 ceph-mon[94241]: osdmap e611: 8 total, 8 up, 8 in 2023-07-12T15:41:02.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:02 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-122"}]': finished 2023-07-12T15:41:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:02 smithi145 ceph-mon[95801]: osdmap e611: 8 total, 8 up, 8 in 2023-07-12T15:41:03.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:03 smithi069 ceph-mon[94241]: osdmap e612: 8 total, 8 up, 8 in 2023-07-12T15:41:03.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:03 smithi069 ceph-mon[94241]: pgmap v931: 172 pgs: 172 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:41:03.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:03 smithi069 ceph-mon[94241]: osdmap e613: 8 total, 8 up, 8 in 2023-07-12T15:41:03.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:03 smithi069 ceph-mon[102777]: osdmap e612: 8 total, 8 up, 8 in 2023-07-12T15:41:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:03 smithi069 ceph-mon[102777]: pgmap v931: 172 pgs: 172 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:41:03.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:03 smithi069 ceph-mon[102777]: osdmap e613: 8 total, 8 up, 8 in 2023-07-12T15:41:03.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:03.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:03 smithi145 ceph-mon[95801]: osdmap e612: 8 total, 8 up, 8 in 2023-07-12T15:41:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:03 smithi145 ceph-mon[95801]: pgmap v931: 172 pgs: 172 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2023-07-12T15:41:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:03 smithi145 ceph-mon[95801]: osdmap e613: 8 total, 8 up, 8 in 2023-07-12T15:41:03.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:05.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-124","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:04 smithi069 ceph-mon[102777]: osdmap e614: 8 total, 8 up, 8 in 2023-07-12T15:41:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:04 smithi069 ceph-mon[102777]: pgmap v934: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:05.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-124","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:04 smithi069 ceph-mon[94241]: osdmap e614: 8 total, 8 up, 8 in 2023-07-12T15:41:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:04 smithi069 ceph-mon[94241]: pgmap v934: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:05.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:41:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:41:05.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-124","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:05.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:04 smithi145 ceph-mon[95801]: osdmap e614: 8 total, 8 up, 8 in 2023-07-12T15:41:05.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:04 smithi145 ceph-mon[95801]: pgmap v934: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:06.236 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:41:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:41:06] "GET /metrics HTTP/1.1" 200 228520 "" "Prometheus/2.43.0" 2023-07-12T15:41:06.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:05 smithi069 ceph-mon[102777]: osdmap e615: 8 total, 8 up, 8 in 2023-07-12T15:41:06.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-124", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:06.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:05 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:05 smithi069 ceph-mon[94241]: osdmap e615: 8 total, 8 up, 8 in 2023-07-12T15:41:06.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-124", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:06.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:05 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:06.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:05 smithi145 ceph-mon[95801]: osdmap e615: 8 total, 8 up, 8 in 2023-07-12T15:41:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-124", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:05 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:07.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-124", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:06 smithi145 ceph-mon[95801]: osdmap e616: 8 total, 8 up, 8 in 2023-07-12T15:41:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-124"}]: dispatch 2023-07-12T15:41:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:06 smithi145 ceph-mon[95801]: pgmap v937: 204 pgs: 204 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:41:07.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-124", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:07.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:06 smithi069 ceph-mon[102777]: osdmap e616: 8 total, 8 up, 8 in 2023-07-12T15:41:07.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-124"}]: dispatch 2023-07-12T15:41:07.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:06 smithi069 ceph-mon[102777]: pgmap v937: 204 pgs: 204 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:41:07.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-124", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:07.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:06 smithi069 ceph-mon[94241]: osdmap e616: 8 total, 8 up, 8 in 2023-07-12T15:41:07.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-124"}]: dispatch 2023-07-12T15:41:07.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:06 smithi069 ceph-mon[94241]: pgmap v937: 204 pgs: 204 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:41:08.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-124"}]': finished 2023-07-12T15:41:08.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:07 smithi145 ceph-mon[95801]: osdmap e617: 8 total, 8 up, 8 in 2023-07-12T15:41:08.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-124", "mode": "writeback"}]: dispatch 2023-07-12T15:41:08.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:07 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:08.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-124", "mode": "writeback"}]': finished 2023-07-12T15:41:08.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:07 smithi145 ceph-mon[95801]: osdmap e618: 8 total, 8 up, 8 in 2023-07-12T15:41:08.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-124"}]': finished 2023-07-12T15:41:08.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:07 smithi069 ceph-mon[102777]: osdmap e617: 8 total, 8 up, 8 in 2023-07-12T15:41:08.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-124", "mode": "writeback"}]: dispatch 2023-07-12T15:41:08.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:07 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:08.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-124", "mode": "writeback"}]': finished 2023-07-12T15:41:08.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:07 smithi069 ceph-mon[102777]: osdmap e618: 8 total, 8 up, 8 in 2023-07-12T15:41:08.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-124"}]': finished 2023-07-12T15:41:08.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:07 smithi069 ceph-mon[94241]: osdmap e617: 8 total, 8 up, 8 in 2023-07-12T15:41:08.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-124", "mode": "writeback"}]: dispatch 2023-07-12T15:41:08.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:07 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:08.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-124", "mode": "writeback"}]': finished 2023-07-12T15:41:08.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:07 smithi069 ceph-mon[94241]: osdmap e618: 8 total, 8 up, 8 in 2023-07-12T15:41:09.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:09 smithi145 ceph-mon[95801]: pgmap v940: 204 pgs: 204 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:41:09.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:09 smithi069 ceph-mon[94241]: pgmap v940: 204 pgs: 204 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:41:09.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:09 smithi069 ceph-mon[102777]: pgmap v940: 204 pgs: 204 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:41:10.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:10 smithi145 ceph-mon[95801]: osdmap e619: 8 total, 8 up, 8 in 2023-07-12T15:41:10.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:10.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:10 smithi069 ceph-mon[94241]: osdmap e619: 8 total, 8 up, 8 in 2023-07-12T15:41:10.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:10 smithi069 ceph-mon[102777]: osdmap e619: 8 total, 8 up, 8 in 2023-07-12T15:41:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:11.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:11 smithi145 ceph-mon[95801]: osdmap e620: 8 total, 8 up, 8 in 2023-07-12T15:41:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-124"}]: dispatch 2023-07-12T15:41:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:11 smithi145 ceph-mon[95801]: pgmap v943: 204 pgs: 204 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2023-07-12T15:41:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:11 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-124"}]': finished 2023-07-12T15:41:11.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:11 smithi145 ceph-mon[95801]: osdmap e621: 8 total, 8 up, 8 in 2023-07-12T15:41:11.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[102777]: osdmap e620: 8 total, 8 up, 8 in 2023-07-12T15:41:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-124"}]: dispatch 2023-07-12T15:41:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[102777]: pgmap v943: 204 pgs: 204 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2023-07-12T15:41:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-124"}]': finished 2023-07-12T15:41:11.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[102777]: osdmap e621: 8 total, 8 up, 8 in 2023-07-12T15:41:11.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:11.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[94241]: osdmap e620: 8 total, 8 up, 8 in 2023-07-12T15:41:11.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-124"}]: dispatch 2023-07-12T15:41:11.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[94241]: pgmap v943: 204 pgs: 204 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2023-07-12T15:41:11.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:11.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-124"}]': finished 2023-07-12T15:41:11.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:11 smithi069 ceph-mon[94241]: osdmap e621: 8 total, 8 up, 8 in 2023-07-12T15:41:13.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:12 smithi069 ceph-mon[102777]: osdmap e622: 8 total, 8 up, 8 in 2023-07-12T15:41:13.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:12 smithi069 ceph-mon[102777]: pgmap v946: 172 pgs: 172 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:41:13.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:12 smithi069 ceph-mon[94241]: osdmap e622: 8 total, 8 up, 8 in 2023-07-12T15:41:13.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:12 smithi069 ceph-mon[94241]: pgmap v946: 172 pgs: 172 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:41:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:12 smithi145 ceph-mon[95801]: osdmap e622: 8 total, 8 up, 8 in 2023-07-12T15:41:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:12 smithi145 ceph-mon[95801]: pgmap v946: 172 pgs: 172 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2023-07-12T15:41:14.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:13 smithi069 ceph-mon[102777]: osdmap e623: 8 total, 8 up, 8 in 2023-07-12T15:41:14.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:14.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:41:14.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-126","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:14.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:13 smithi069 ceph-mon[102777]: osdmap e624: 8 total, 8 up, 8 in 2023-07-12T15:41:14.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:13 smithi069 ceph-mon[94241]: osdmap e623: 8 total, 8 up, 8 in 2023-07-12T15:41:14.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:14.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:41:14.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-126","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:14.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:13 smithi069 ceph-mon[94241]: osdmap e624: 8 total, 8 up, 8 in 2023-07-12T15:41:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:13 smithi145 ceph-mon[95801]: osdmap e623: 8 total, 8 up, 8 in 2023-07-12T15:41:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:41:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-126","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:13 smithi145 ceph-mon[95801]: osdmap e624: 8 total, 8 up, 8 in 2023-07-12T15:41:15.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-126", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:15.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:14 smithi069 ceph-mon[102777]: pgmap v949: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:15.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-126", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:15.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:14 smithi069 ceph-mon[94241]: pgmap v949: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:15.352 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:41:15 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:41:15.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-126", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:15.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:14 smithi145 ceph-mon[95801]: pgmap v949: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 663 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:16.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:41:16] "GET /metrics HTTP/1.1" 200 228534 "" "Prometheus/2.43.0" 2023-07-12T15:41:16.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-126", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:16.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:15 smithi069 ceph-mon[94241]: osdmap e625: 8 total, 8 up, 8 in 2023-07-12T15:41:16.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-126"}]: dispatch 2023-07-12T15:41:16.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:15 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:16.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-126", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:16.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:15 smithi069 ceph-mon[102777]: osdmap e625: 8 total, 8 up, 8 in 2023-07-12T15:41:16.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-126"}]: dispatch 2023-07-12T15:41:16.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:15 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:16.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-126", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:15 smithi145 ceph-mon[95801]: osdmap e625: 8 total, 8 up, 8 in 2023-07-12T15:41:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-126"}]: dispatch 2023-07-12T15:41:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:15 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:17.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-126"}]': finished 2023-07-12T15:41:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[102777]: osdmap e626: 8 total, 8 up, 8 in 2023-07-12T15:41:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-126", "mode": "writeback"}]: dispatch 2023-07-12T15:41:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[102777]: pgmap v952: 204 pgs: 204 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:17.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-126", "mode": "writeback"}]': finished 2023-07-12T15:41:17.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[102777]: osdmap e627: 8 total, 8 up, 8 in 2023-07-12T15:41:17.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-126"}]': finished 2023-07-12T15:41:17.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[94241]: osdmap e626: 8 total, 8 up, 8 in 2023-07-12T15:41:17.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-126", "mode": "writeback"}]: dispatch 2023-07-12T15:41:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[94241]: pgmap v952: 204 pgs: 204 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-126", "mode": "writeback"}]': finished 2023-07-12T15:41:17.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:16 smithi069 ceph-mon[94241]: osdmap e627: 8 total, 8 up, 8 in 2023-07-12T15:41:17.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-126"}]': finished 2023-07-12T15:41:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:16 smithi145 ceph-mon[95801]: osdmap e626: 8 total, 8 up, 8 in 2023-07-12T15:41:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-126", "mode": "writeback"}]: dispatch 2023-07-12T15:41:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:16 smithi145 ceph-mon[95801]: pgmap v952: 204 pgs: 204 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:17.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:16 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:17.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-126", "mode": "writeback"}]': finished 2023-07-12T15:41:17.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:16 smithi145 ceph-mon[95801]: osdmap e627: 8 total, 8 up, 8 in 2023-07-12T15:41:18.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:18.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:19.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:19.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:18 smithi069 ceph-mon[102777]: osdmap e628: 8 total, 8 up, 8 in 2023-07-12T15:41:19.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-126"}]: dispatch 2023-07-12T15:41:19.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:18 smithi069 ceph-mon[102777]: pgmap v955: 204 pgs: 204 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:19.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:18 smithi069 ceph-mon[94241]: osdmap e628: 8 total, 8 up, 8 in 2023-07-12T15:41:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-126"}]: dispatch 2023-07-12T15:41:19.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:18 smithi069 ceph-mon[94241]: pgmap v955: 204 pgs: 204 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:19.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:18 smithi145 ceph-mon[95801]: osdmap e628: 8 total, 8 up, 8 in 2023-07-12T15:41:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-126"}]: dispatch 2023-07-12T15:41:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:18 smithi145 ceph-mon[95801]: pgmap v955: 204 pgs: 204 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:19.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:41:19.969 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9064 ms) 2023-07-12T15:41:19.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2023-07-12T15:41:19.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:41:19.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6057 ms) 2023-07-12T15:41:19.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2023-07-12T15:41:19.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:41:19.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5050 ms) 2023-07-12T15:41:19.970 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2023-07-12T15:41:19.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:41:19.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26013 ms) 2023-07-12T15:41:19.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2023-07-12T15:41:19.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:41:19.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13652 ms) 2023-07-12T15:41:19.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2023-07-12T15:41:19.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12082 ms) 2023-07-12T15:41:19.971 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2023-07-12T15:41:19.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2023-07-12T15:41:19.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000010 69 73 |is| 2023-07-12T15:41:19.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000012 2023-07-12T15:41:19.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12088 ms) 2023-07-12T15:41:19.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2023-07-12T15:41:19.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = reef 2023-07-12T15:41:19.972 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6035 ms) 2023-07-12T15:41:19.973 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (553939 ms total) 2023-07-12T15:41:19.973 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2023-07-12T15:41:19.973 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2023-07-12T15:41:19.973 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2023-07-12T15:41:19.973 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1019 ms) 2023-07-12T15:41:19.973 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2023-07-12T15:41:19.973 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11057 ms) 2023-07-12T15:41:19.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2023-07-12T15:41:19.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18126 ms) 2023-07-12T15:41:19.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2023-07-12T15:41:19.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2023-07-12T15:41:19.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30204 ms total) 2023-07-12T15:41:19.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2023-07-12T15:41:19.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2023-07-12T15:41:19.974 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2023-07-12T15:41:19.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7046 ms) 2023-07-12T15:41:19.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2023-07-12T15:41:19.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8053 ms) 2023-07-12T15:41:19.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2023-07-12T15:41:19.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrub... 2023-07-12T15:41:19.975 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2023-07-12T15:41:19.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24458 ms) 2023-07-12T15:41:19.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2023-07-12T15:41:19.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10034 ms) 2023-07-12T15:41:19.976 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2023-07-12T15:41:19.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7951 ms) 2023-07-12T15:41:19.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2023-07-12T15:41:19.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8053 ms) 2023-07-12T15:41:19.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2023-07-12T15:41:19.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (9948 ms) 2023-07-12T15:41:19.977 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2023-07-12T15:41:20.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:19 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:20.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-126"}]': finished 2023-07-12T15:41:20.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:19 smithi069 ceph-mon[94241]: osdmap e629: 8 total, 8 up, 8 in 2023-07-12T15:41:20.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:19 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:20.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-126"}]': finished 2023-07-12T15:41:20.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:19 smithi069 ceph-mon[102777]: osdmap e629: 8 total, 8 up, 8 in 2023-07-12T15:41:20.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:19 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:20.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-126"}]': finished 2023-07-12T15:41:20.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:19 smithi145 ceph-mon[95801]: osdmap e629: 8 total, 8 up, 8 in 2023-07-12T15:41:21.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:20 smithi069 ceph-mon[102777]: osdmap e630: 8 total, 8 up, 8 in 2023-07-12T15:41:21.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:20 smithi069 ceph-mon[102777]: pgmap v958: 172 pgs: 172 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:41:21.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:20 smithi069 ceph-mon[94241]: osdmap e630: 8 total, 8 up, 8 in 2023-07-12T15:41:21.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:20 smithi069 ceph-mon[94241]: pgmap v958: 172 pgs: 172 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:41:21.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:20 smithi145 ceph-mon[95801]: osdmap e630: 8 total, 8 up, 8 in 2023-07-12T15:41:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:20 smithi145 ceph-mon[95801]: pgmap v958: 172 pgs: 172 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:41:22.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:21 smithi145 ceph-mon[95801]: osdmap e631: 8 total, 8 up, 8 in 2023-07-12T15:41:22.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:22.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:21 smithi069 ceph-mon[94241]: osdmap e631: 8 total, 8 up, 8 in 2023-07-12T15:41:22.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:22.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:21 smithi069 ceph-mon[102777]: osdmap e631: 8 total, 8 up, 8 in 2023-07-12T15:41:22.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:23.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-128","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:22 smithi145 ceph-mon[95801]: osdmap e632: 8 total, 8 up, 8 in 2023-07-12T15:41:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-128", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:22 smithi145 ceph-mon[95801]: pgmap v961: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:41:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-128", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:22 smithi145 ceph-mon[95801]: osdmap e633: 8 total, 8 up, 8 in 2023-07-12T15:41:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:22 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-128"}]: dispatch 2023-07-12T15:41:23.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-128","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[94241]: osdmap e632: 8 total, 8 up, 8 in 2023-07-12T15:41:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-128", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[94241]: pgmap v961: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:41:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-128", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[94241]: osdmap e633: 8 total, 8 up, 8 in 2023-07-12T15:41:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-128"}]: dispatch 2023-07-12T15:41:23.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-128","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:23.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[102777]: osdmap e632: 8 total, 8 up, 8 in 2023-07-12T15:41:23.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-128", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:23.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[102777]: pgmap v961: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:41:23.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-128", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:23.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[102777]: osdmap e633: 8 total, 8 up, 8 in 2023-07-12T15:41:23.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:22 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-128"}]: dispatch 2023-07-12T15:41:25.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-128"}]': finished 2023-07-12T15:41:25.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:24 smithi069 ceph-mon[102777]: osdmap e634: 8 total, 8 up, 8 in 2023-07-12T15:41:25.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-128", "mode": "writeback"}]: dispatch 2023-07-12T15:41:25.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:24 smithi069 ceph-mon[102777]: pgmap v964: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:25.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-128"}]': finished 2023-07-12T15:41:25.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:24 smithi069 ceph-mon[94241]: osdmap e634: 8 total, 8 up, 8 in 2023-07-12T15:41:25.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-128", "mode": "writeback"}]: dispatch 2023-07-12T15:41:25.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:24 smithi069 ceph-mon[94241]: pgmap v964: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:25.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:41:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:41:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-128"}]': finished 2023-07-12T15:41:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:24 smithi145 ceph-mon[95801]: osdmap e634: 8 total, 8 up, 8 in 2023-07-12T15:41:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-128", "mode": "writeback"}]: dispatch 2023-07-12T15:41:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:24 smithi145 ceph-mon[95801]: pgmap v964: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:26.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:41:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:41:26] "GET /metrics HTTP/1.1" 200 228534 "" "Prometheus/2.43.0" 2023-07-12T15:41:26.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:25 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:26.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-128", "mode": "writeback"}]': finished 2023-07-12T15:41:26.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:25 smithi069 ceph-mon[94241]: osdmap e635: 8 total, 8 up, 8 in 2023-07-12T15:41:26.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:25 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:26.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:25 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:41:26.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:25 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:26.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-128", "mode": "writeback"}]': finished 2023-07-12T15:41:26.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:25 smithi069 ceph-mon[102777]: osdmap e635: 8 total, 8 up, 8 in 2023-07-12T15:41:26.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:25 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:26.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:25 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:41:26.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:25 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:26.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-128", "mode": "writeback"}]': finished 2023-07-12T15:41:26.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:25 smithi145 ceph-mon[95801]: osdmap e635: 8 total, 8 up, 8 in 2023-07-12T15:41:26.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:25 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:26.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:25 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:41:27.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:26 smithi145 ceph-mon[95801]: pgmap v966: 204 pgs: 204 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:27.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:26 smithi069 ceph-mon[94241]: pgmap v966: 204 pgs: 204 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:27.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:26 smithi069 ceph-mon[102777]: pgmap v966: 204 pgs: 204 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:29.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:29 smithi069 ceph-mon[102777]: pgmap v967: 204 pgs: 204 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:41:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:29 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:41:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:29 smithi069 ceph-mon[94241]: pgmap v967: 204 pgs: 204 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:41:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:29 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:41:29.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:29 smithi145 ceph-mon[95801]: pgmap v967: 204 pgs: 204 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:41:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:29 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:41:30.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:30 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:30.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:30 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:30.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:30 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:30.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:30 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:30.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:30 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:30.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:30 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:30.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:30 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:30.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:30 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:30.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:30 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:30.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:30 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:31.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:30 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:31.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:30 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:31.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:30 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:31.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:30 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:31.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:30 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:31.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[102777]: pgmap v968: 204 pgs: 204 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 283 B/s wr, 1 op/s 2023-07-12T15:41:31.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:31.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[102777]: osdmap e636: 8 total, 8 up, 8 in 2023-07-12T15:41:31.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-128"}]: dispatch 2023-07-12T15:41:31.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:41:31.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:41:31.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:31.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[94241]: pgmap v968: 204 pgs: 204 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 283 B/s wr, 1 op/s 2023-07-12T15:41:31.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:31.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[94241]: osdmap e636: 8 total, 8 up, 8 in 2023-07-12T15:41:31.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-128"}]: dispatch 2023-07-12T15:41:31.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:41:31.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:41:31.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:31 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:32.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:31 smithi145 ceph-mon[95801]: pgmap v968: 204 pgs: 204 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 283 B/s wr, 1 op/s 2023-07-12T15:41:32.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:32.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:31 smithi145 ceph-mon[95801]: osdmap e636: 8 total, 8 up, 8 in 2023-07-12T15:41:32.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-128"}]: dispatch 2023-07-12T15:41:32.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:31 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:41:32.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:31 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:41:32.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:31 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:32.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:32 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:32.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-128"}]': finished 2023-07-12T15:41:32.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:32 smithi069 ceph-mon[102777]: osdmap e637: 8 total, 8 up, 8 in 2023-07-12T15:41:32.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:32 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:32.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-128"}]': finished 2023-07-12T15:41:32.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:32 smithi069 ceph-mon[94241]: osdmap e637: 8 total, 8 up, 8 in 2023-07-12T15:41:33.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:32 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:33.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-128"}]': finished 2023-07-12T15:41:33.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:32 smithi145 ceph-mon[95801]: osdmap e637: 8 total, 8 up, 8 in 2023-07-12T15:41:33.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:33 smithi069 ceph-mon[102777]: pgmap v971: 204 pgs: 204 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 284 B/s wr, 1 op/s 2023-07-12T15:41:33.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:33 smithi069 ceph-mon[102777]: osdmap e638: 8 total, 8 up, 8 in 2023-07-12T15:41:33.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:33 smithi069 ceph-mon[102777]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:41:33.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:33 smithi069 ceph-mon[94241]: pgmap v971: 204 pgs: 204 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 284 B/s wr, 1 op/s 2023-07-12T15:41:33.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:33 smithi069 ceph-mon[94241]: osdmap e638: 8 total, 8 up, 8 in 2023-07-12T15:41:33.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:33 smithi069 ceph-mon[94241]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:41:34.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:33 smithi145 ceph-mon[95801]: pgmap v971: 204 pgs: 204 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 284 B/s wr, 1 op/s 2023-07-12T15:41:34.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:33 smithi145 ceph-mon[95801]: osdmap e638: 8 total, 8 up, 8 in 2023-07-12T15:41:34.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:33 smithi145 ceph-mon[95801]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:41:35.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:34 smithi145 ceph-mon[95801]: osdmap e639: 8 total, 8 up, 8 in 2023-07-12T15:41:35.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:35.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:34 smithi145 ceph-mon[95801]: pgmap v974: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:35.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:34 smithi069 ceph-mon[102777]: osdmap e639: 8 total, 8 up, 8 in 2023-07-12T15:41:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:34 smithi069 ceph-mon[102777]: pgmap v974: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:35.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:34 smithi069 ceph-mon[94241]: osdmap e639: 8 total, 8 up, 8 in 2023-07-12T15:41:35.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:35.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:34 smithi069 ceph-mon[94241]: pgmap v974: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:35.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:41:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:41:36.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-130","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:35 smithi145 ceph-mon[95801]: osdmap e640: 8 total, 8 up, 8 in 2023-07-12T15:41:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-130", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:36.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:35 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:36.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-130","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:36.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:35 smithi069 ceph-mon[102777]: osdmap e640: 8 total, 8 up, 8 in 2023-07-12T15:41:36.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-130", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:36.150 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:35 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:36.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-130","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:36.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:35 smithi069 ceph-mon[94241]: osdmap e640: 8 total, 8 up, 8 in 2023-07-12T15:41:36.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-130", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:36.151 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:35 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:36.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:41:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:41:36] "GET /metrics HTTP/1.1" 200 228534 "" "Prometheus/2.43.0" 2023-07-12T15:41:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-130", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:36 smithi145 ceph-mon[95801]: osdmap e641: 8 total, 8 up, 8 in 2023-07-12T15:41:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-130"}]: dispatch 2023-07-12T15:41:37.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:36 smithi145 ceph-mon[95801]: pgmap v977: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:41:37.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-130", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:37.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:36 smithi069 ceph-mon[102777]: osdmap e641: 8 total, 8 up, 8 in 2023-07-12T15:41:37.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-130"}]: dispatch 2023-07-12T15:41:37.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:36 smithi069 ceph-mon[102777]: pgmap v977: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:41:37.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-130", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:37.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:36 smithi069 ceph-mon[94241]: osdmap e641: 8 total, 8 up, 8 in 2023-07-12T15:41:37.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-130"}]: dispatch 2023-07-12T15:41:37.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:36 smithi069 ceph-mon[94241]: pgmap v977: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:41:38.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-130"}]': finished 2023-07-12T15:41:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:37 smithi145 ceph-mon[95801]: osdmap e642: 8 total, 8 up, 8 in 2023-07-12T15:41:38.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-130", "mode": "writeback"}]: dispatch 2023-07-12T15:41:38.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-130"}]': finished 2023-07-12T15:41:38.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:37 smithi069 ceph-mon[94241]: osdmap e642: 8 total, 8 up, 8 in 2023-07-12T15:41:38.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-130", "mode": "writeback"}]: dispatch 2023-07-12T15:41:38.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-130"}]': finished 2023-07-12T15:41:38.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:37 smithi069 ceph-mon[102777]: osdmap e642: 8 total, 8 up, 8 in 2023-07-12T15:41:38.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-130", "mode": "writeback"}]: dispatch 2023-07-12T15:41:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:38 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-130", "mode": "writeback"}]': finished 2023-07-12T15:41:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:38 smithi145 ceph-mon[95801]: osdmap e643: 8 total, 8 up, 8 in 2023-07-12T15:41:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:39.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:38 smithi145 ceph-mon[95801]: pgmap v980: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:39.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:38 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:39.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-130", "mode": "writeback"}]': finished 2023-07-12T15:41:39.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:38 smithi069 ceph-mon[94241]: osdmap e643: 8 total, 8 up, 8 in 2023-07-12T15:41:39.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:39.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:38 smithi069 ceph-mon[94241]: pgmap v980: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:38 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-130", "mode": "writeback"}]': finished 2023-07-12T15:41:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:38 smithi069 ceph-mon[102777]: osdmap e643: 8 total, 8 up, 8 in 2023-07-12T15:41:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:39.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:38 smithi069 ceph-mon[102777]: pgmap v980: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:40.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:40.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:39 smithi145 ceph-mon[95801]: osdmap e644: 8 total, 8 up, 8 in 2023-07-12T15:41:40.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-130"}]: dispatch 2023-07-12T15:41:40.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:40.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:39 smithi069 ceph-mon[94241]: osdmap e644: 8 total, 8 up, 8 in 2023-07-12T15:41:40.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-130"}]: dispatch 2023-07-12T15:41:40.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:40.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:39 smithi069 ceph-mon[102777]: osdmap e644: 8 total, 8 up, 8 in 2023-07-12T15:41:40.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-130"}]: dispatch 2023-07-12T15:41:41.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:40 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:41.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-130"}]': finished 2023-07-12T15:41:41.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:40 smithi145 ceph-mon[95801]: osdmap e645: 8 total, 8 up, 8 in 2023-07-12T15:41:41.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:40 smithi145 ceph-mon[95801]: pgmap v983: 204 pgs: 204 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2023-07-12T15:41:41.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:40 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:41.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-130"}]': finished 2023-07-12T15:41:41.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:40 smithi069 ceph-mon[102777]: osdmap e645: 8 total, 8 up, 8 in 2023-07-12T15:41:41.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:40 smithi069 ceph-mon[102777]: pgmap v983: 204 pgs: 204 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2023-07-12T15:41:41.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:40 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:41.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-130"}]': finished 2023-07-12T15:41:41.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:40 smithi069 ceph-mon[94241]: osdmap e645: 8 total, 8 up, 8 in 2023-07-12T15:41:41.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:40 smithi069 ceph-mon[94241]: pgmap v983: 204 pgs: 204 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2023-07-12T15:41:42.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:41 smithi145 ceph-mon[95801]: osdmap e646: 8 total, 8 up, 8 in 2023-07-12T15:41:42.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:41 smithi069 ceph-mon[102777]: osdmap e646: 8 total, 8 up, 8 in 2023-07-12T15:41:42.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:41 smithi069 ceph-mon[94241]: osdmap e646: 8 total, 8 up, 8 in 2023-07-12T15:41:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:42 smithi145 ceph-mon[95801]: osdmap e647: 8 total, 8 up, 8 in 2023-07-12T15:41:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:42 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:42 smithi145 ceph-mon[95801]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:41:43.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:42 smithi145 ceph-mon[95801]: pgmap v986: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:41:43.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:42 smithi069 ceph-mon[102777]: osdmap e647: 8 total, 8 up, 8 in 2023-07-12T15:41:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:42 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:42 smithi069 ceph-mon[102777]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:41:43.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:42 smithi069 ceph-mon[102777]: pgmap v986: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:41:43.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:42 smithi069 ceph-mon[94241]: osdmap e647: 8 total, 8 up, 8 in 2023-07-12T15:41:43.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:42 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:43.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:42 smithi069 ceph-mon[94241]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:41:43.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:42 smithi069 ceph-mon[94241]: pgmap v986: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2023-07-12T15:41:44.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-132","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:43 smithi145 ceph-mon[95801]: osdmap e648: 8 total, 8 up, 8 in 2023-07-12T15:41:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-132", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:43 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:44.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:43 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:41:44.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-132","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:43 smithi069 ceph-mon[102777]: osdmap e648: 8 total, 8 up, 8 in 2023-07-12T15:41:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-132", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:43 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:44.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:43 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:41:44.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-132","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:43 smithi069 ceph-mon[94241]: osdmap e648: 8 total, 8 up, 8 in 2023-07-12T15:41:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-132", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:43 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:41:44.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:43 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:41:45.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-132", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:44 smithi145 ceph-mon[95801]: osdmap e649: 8 total, 8 up, 8 in 2023-07-12T15:41:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-132"}]: dispatch 2023-07-12T15:41:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:44 smithi145 ceph-mon[95801]: pgmap v989: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-132", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:44 smithi069 ceph-mon[102777]: osdmap e649: 8 total, 8 up, 8 in 2023-07-12T15:41:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-132"}]: dispatch 2023-07-12T15:41:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:44 smithi069 ceph-mon[102777]: pgmap v989: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-132", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:44 smithi069 ceph-mon[94241]: osdmap e649: 8 total, 8 up, 8 in 2023-07-12T15:41:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-132"}]: dispatch 2023-07-12T15:41:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:44 smithi069 ceph-mon[94241]: pgmap v989: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 670 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:45.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:41:45 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:41:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-132"}]': finished 2023-07-12T15:41:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:45 smithi145 ceph-mon[95801]: osdmap e650: 8 total, 8 up, 8 in 2023-07-12T15:41:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-132", "mode": "writeback"}]: dispatch 2023-07-12T15:41:46.102 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:45 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:46.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-132"}]': finished 2023-07-12T15:41:46.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:45 smithi069 ceph-mon[102777]: osdmap e650: 8 total, 8 up, 8 in 2023-07-12T15:41:46.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-132", "mode": "writeback"}]: dispatch 2023-07-12T15:41:46.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:45 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:46.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-132"}]': finished 2023-07-12T15:41:46.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:45 smithi069 ceph-mon[94241]: osdmap e650: 8 total, 8 up, 8 in 2023-07-12T15:41:46.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-132", "mode": "writeback"}]: dispatch 2023-07-12T15:41:46.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:45 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:46.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:41:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:41:46] "GET /metrics HTTP/1.1" 200 228532 "" "Prometheus/2.43.0" 2023-07-12T15:41:47.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:46 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:47.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-132", "mode": "writeback"}]': finished 2023-07-12T15:41:47.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:46 smithi145 ceph-mon[95801]: osdmap e651: 8 total, 8 up, 8 in 2023-07-12T15:41:47.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:46 smithi145 ceph-mon[95801]: pgmap v992: 204 pgs: 204 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:47.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:46 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-132", "mode": "writeback"}]': finished 2023-07-12T15:41:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:46 smithi069 ceph-mon[94241]: osdmap e651: 8 total, 8 up, 8 in 2023-07-12T15:41:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:46 smithi069 ceph-mon[94241]: pgmap v992: 204 pgs: 204 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:47.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:46 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:47.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-132", "mode": "writeback"}]': finished 2023-07-12T15:41:47.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:46 smithi069 ceph-mon[102777]: osdmap e651: 8 total, 8 up, 8 in 2023-07-12T15:41:47.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:46 smithi069 ceph-mon[102777]: pgmap v992: 204 pgs: 204 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:48.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:47 smithi069 ceph-mon[102777]: osdmap e652: 8 total, 8 up, 8 in 2023-07-12T15:41:48.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:47 smithi069 ceph-mon[94241]: osdmap e652: 8 total, 8 up, 8 in 2023-07-12T15:41:48.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:47 smithi145 ceph-mon[95801]: osdmap e652: 8 total, 8 up, 8 in 2023-07-12T15:41:49.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:48 smithi069 ceph-mon[102777]: osdmap e653: 8 total, 8 up, 8 in 2023-07-12T15:41:49.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:49.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:48 smithi069 ceph-mon[102777]: pgmap v995: 204 pgs: 204 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:49.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:48 smithi069 ceph-mon[94241]: osdmap e653: 8 total, 8 up, 8 in 2023-07-12T15:41:49.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:49.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:48 smithi069 ceph-mon[94241]: pgmap v995: 204 pgs: 204 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:48 smithi145 ceph-mon[95801]: osdmap e653: 8 total, 8 up, 8 in 2023-07-12T15:41:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:49.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:48 smithi145 ceph-mon[95801]: pgmap v995: 204 pgs: 204 active+clean; 582 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:50.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:50.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:49 smithi069 ceph-mon[102777]: osdmap e654: 8 total, 8 up, 8 in 2023-07-12T15:41:50.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-132"}]: dispatch 2023-07-12T15:41:50.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:50.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:49 smithi069 ceph-mon[94241]: osdmap e654: 8 total, 8 up, 8 in 2023-07-12T15:41:50.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-132"}]: dispatch 2023-07-12T15:41:50.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:50.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:49 smithi145 ceph-mon[95801]: osdmap e654: 8 total, 8 up, 8 in 2023-07-12T15:41:50.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-132"}]: dispatch 2023-07-12T15:41:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-132"}]': finished 2023-07-12T15:41:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:50 smithi069 ceph-mon[102777]: osdmap e655: 8 total, 8 up, 8 in 2023-07-12T15:41:51.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:50 smithi069 ceph-mon[102777]: pgmap v998: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-07-12T15:41:51.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-132"}]': finished 2023-07-12T15:41:51.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:50 smithi069 ceph-mon[94241]: osdmap e655: 8 total, 8 up, 8 in 2023-07-12T15:41:51.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:50 smithi069 ceph-mon[94241]: pgmap v998: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-07-12T15:41:51.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-132"}]': finished 2023-07-12T15:41:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:50 smithi145 ceph-mon[95801]: osdmap e655: 8 total, 8 up, 8 in 2023-07-12T15:41:51.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:50 smithi145 ceph-mon[95801]: pgmap v998: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-07-12T15:41:52.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:51 smithi069 ceph-mon[102777]: osdmap e656: 8 total, 8 up, 8 in 2023-07-12T15:41:52.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:52.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:51 smithi069 ceph-mon[94241]: osdmap e656: 8 total, 8 up, 8 in 2023-07-12T15:41:52.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:51 smithi145 ceph-mon[95801]: osdmap e656: 8 total, 8 up, 8 in 2023-07-12T15:41:52.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:41:53.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[94241]: osdmap e657: 8 total, 8 up, 8 in 2023-07-12T15:41:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-132"}]: dispatch 2023-07-12T15:41:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[94241]: pgmap v1001: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-07-12T15:41:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-132"}]': finished 2023-07-12T15:41:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[94241]: osdmap e658: 8 total, 8 up, 8 in 2023-07-12T15:41:53.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:53.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[102777]: osdmap e657: 8 total, 8 up, 8 in 2023-07-12T15:41:53.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-132"}]: dispatch 2023-07-12T15:41:53.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[102777]: pgmap v1001: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-07-12T15:41:53.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:53.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-132"}]': finished 2023-07-12T15:41:53.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:52 smithi069 ceph-mon[102777]: osdmap e658: 8 total, 8 up, 8 in 2023-07-12T15:41:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:41:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:52 smithi145 ceph-mon[95801]: osdmap e657: 8 total, 8 up, 8 in 2023-07-12T15:41:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-132"}]: dispatch 2023-07-12T15:41:53.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:52 smithi145 ceph-mon[95801]: pgmap v1001: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-07-12T15:41:53.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:52 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:41:53.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-132"}]': finished 2023-07-12T15:41:53.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:52 smithi145 ceph-mon[95801]: osdmap e658: 8 total, 8 up, 8 in 2023-07-12T15:41:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:54 smithi069 ceph-mon[102777]: osdmap e659: 8 total, 8 up, 8 in 2023-07-12T15:41:55.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:54 smithi069 ceph-mon[102777]: pgmap v1004: 172 pgs: 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:55.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:54 smithi069 ceph-mon[94241]: osdmap e659: 8 total, 8 up, 8 in 2023-07-12T15:41:55.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:54 smithi069 ceph-mon[94241]: pgmap v1004: 172 pgs: 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:55.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:41:55 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:41:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:54 smithi145 ceph-mon[95801]: osdmap e659: 8 total, 8 up, 8 in 2023-07-12T15:41:55.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:54 smithi145 ceph-mon[95801]: pgmap v1004: 172 pgs: 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:41:56.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:41:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:41:56] "GET /metrics HTTP/1.1" 200 228532 "" "Prometheus/2.43.0" 2023-07-12T15:41:56.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:55 smithi069 ceph-mon[94241]: osdmap e660: 8 total, 8 up, 8 in 2023-07-12T15:41:56.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:56.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:55 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:56.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:55 smithi069 ceph-mon[102777]: osdmap e660: 8 total, 8 up, 8 in 2023-07-12T15:41:56.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:56.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:55 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:56.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:55 smithi145 ceph-mon[95801]: osdmap e660: 8 total, 8 up, 8 in 2023-07-12T15:41:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:41:56.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:55 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:41:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-134","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:57.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:56 smithi069 ceph-mon[102777]: osdmap e661: 8 total, 8 up, 8 in 2023-07-12T15:41:57.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-134", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:57.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:56 smithi069 ceph-mon[102777]: pgmap v1007: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2023-07-12T15:41:57.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-134","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:57.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:56 smithi069 ceph-mon[94241]: osdmap e661: 8 total, 8 up, 8 in 2023-07-12T15:41:57.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-134", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:57.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:56 smithi069 ceph-mon[94241]: pgmap v1007: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2023-07-12T15:41:57.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-134","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:41:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:56 smithi145 ceph-mon[95801]: osdmap e661: 8 total, 8 up, 8 in 2023-07-12T15:41:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-134", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:41:57.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:56 smithi145 ceph-mon[95801]: pgmap v1007: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2023-07-12T15:41:58.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-134", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:58.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:57 smithi069 ceph-mon[102777]: osdmap e662: 8 total, 8 up, 8 in 2023-07-12T15:41:58.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-134"}]: dispatch 2023-07-12T15:41:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-134", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:57 smithi069 ceph-mon[94241]: osdmap e662: 8 total, 8 up, 8 in 2023-07-12T15:41:58.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-134"}]: dispatch 2023-07-12T15:41:58.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-134", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:41:58.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:57 smithi145 ceph-mon[95801]: osdmap e662: 8 total, 8 up, 8 in 2023-07-12T15:41:58.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-134"}]: dispatch 2023-07-12T15:41:59.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-134"}]': finished 2023-07-12T15:41:59.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[94241]: osdmap e663: 8 total, 8 up, 8 in 2023-07-12T15:41:59.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-134", "mode": "writeback"}]: dispatch 2023-07-12T15:41:59.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[94241]: pgmap v1010: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:59.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:41:59.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:59.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-134", "mode": "writeback"}]': finished 2023-07-12T15:41:59.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[94241]: osdmap e664: 8 total, 8 up, 8 in 2023-07-12T15:41:59.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-134"}]': finished 2023-07-12T15:41:59.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[102777]: osdmap e663: 8 total, 8 up, 8 in 2023-07-12T15:41:59.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-134", "mode": "writeback"}]: dispatch 2023-07-12T15:41:59.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[102777]: pgmap v1010: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:59.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:41:59.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:59.240 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-134", "mode": "writeback"}]': finished 2023-07-12T15:41:59.241 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:58 smithi069 ceph-mon[102777]: osdmap e664: 8 total, 8 up, 8 in 2023-07-12T15:41:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-134"}]': finished 2023-07-12T15:41:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:58 smithi145 ceph-mon[95801]: osdmap e663: 8 total, 8 up, 8 in 2023-07-12T15:41:59.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-134", "mode": "writeback"}]: dispatch 2023-07-12T15:41:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:58 smithi145 ceph-mon[95801]: pgmap v1010: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:41:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:58 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:41:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:58 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:41:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-134", "mode": "writeback"}]': finished 2023-07-12T15:41:59.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:58 smithi145 ceph-mon[95801]: osdmap e664: 8 total, 8 up, 8 in 2023-07-12T15:42:00.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:41:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:00.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:41:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:00.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:41:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:42:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:00 smithi069 ceph-mon[102777]: osdmap e665: 8 total, 8 up, 8 in 2023-07-12T15:42:01.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:00 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-134"}]: dispatch 2023-07-12T15:42:01.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:00 smithi069 ceph-mon[102777]: pgmap v1013: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2023-07-12T15:42:01.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:42:01.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:00 smithi069 ceph-mon[94241]: osdmap e665: 8 total, 8 up, 8 in 2023-07-12T15:42:01.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:00 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-134"}]: dispatch 2023-07-12T15:42:01.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:00 smithi069 ceph-mon[94241]: pgmap v1013: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2023-07-12T15:42:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:42:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:00 smithi145 ceph-mon[95801]: osdmap e665: 8 total, 8 up, 8 in 2023-07-12T15:42:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:00 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-134"}]: dispatch 2023-07-12T15:42:01.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:00 smithi145 ceph-mon[95801]: pgmap v1013: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2023-07-12T15:42:02.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:02 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:42:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:02 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-134"}]': finished 2023-07-12T15:42:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:02 smithi069 ceph-mon[102777]: osdmap e666: 8 total, 8 up, 8 in 2023-07-12T15:42:02.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:02 smithi069 ceph-mon[102777]: osdmap e667: 8 total, 8 up, 8 in 2023-07-12T15:42:02.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:02 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:42:02.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:02 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-134"}]': finished 2023-07-12T15:42:02.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:02 smithi069 ceph-mon[94241]: osdmap e666: 8 total, 8 up, 8 in 2023-07-12T15:42:02.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:02 smithi069 ceph-mon[94241]: osdmap e667: 8 total, 8 up, 8 in 2023-07-12T15:42:02.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:02 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:42:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:02 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-134"}]': finished 2023-07-12T15:42:02.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:02 smithi145 ceph-mon[95801]: osdmap e666: 8 total, 8 up, 8 in 2023-07-12T15:42:02.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:02 smithi145 ceph-mon[95801]: osdmap e667: 8 total, 8 up, 8 in 2023-07-12T15:42:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:03 smithi069 ceph-mon[102777]: pgmap v1016: 172 pgs: 172 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:42:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:03 smithi069 ceph-mon[102777]: osdmap e668: 8 total, 8 up, 8 in 2023-07-12T15:42:03.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:03 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:03 smithi069 ceph-mon[94241]: pgmap v1016: 172 pgs: 172 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:42:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:03 smithi069 ceph-mon[94241]: osdmap e668: 8 total, 8 up, 8 in 2023-07-12T15:42:03.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:03 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:03.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:03 smithi145 ceph-mon[95801]: pgmap v1016: 172 pgs: 172 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-07-12T15:42:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:03 smithi145 ceph-mon[95801]: osdmap e668: 8 total, 8 up, 8 in 2023-07-12T15:42:03.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:03 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-136","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:04 smithi069 ceph-mon[102777]: osdmap e669: 8 total, 8 up, 8 in 2023-07-12T15:42:05.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-136", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:05.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:04 smithi069 ceph-mon[102777]: pgmap v1019: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-136","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:04 smithi069 ceph-mon[94241]: osdmap e669: 8 total, 8 up, 8 in 2023-07-12T15:42:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-136", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:05.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:04 smithi069 ceph-mon[94241]: pgmap v1019: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:05.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:42:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:42:05.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-136","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:05.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:04 smithi145 ceph-mon[95801]: osdmap e669: 8 total, 8 up, 8 in 2023-07-12T15:42:05.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-136", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:05.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:04 smithi145 ceph-mon[95801]: pgmap v1019: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:06.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:42:06] "GET /metrics HTTP/1.1" 200 228568 "" "Prometheus/2.43.0" 2023-07-12T15:42:06.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-136", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:06.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:05 smithi069 ceph-mon[94241]: osdmap e670: 8 total, 8 up, 8 in 2023-07-12T15:42:06.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-136"}]: dispatch 2023-07-12T15:42:06.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:05 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:06.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-136", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:06.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:05 smithi069 ceph-mon[102777]: osdmap e670: 8 total, 8 up, 8 in 2023-07-12T15:42:06.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-136"}]: dispatch 2023-07-12T15:42:06.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:05 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-136", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:05 smithi145 ceph-mon[95801]: osdmap e670: 8 total, 8 up, 8 in 2023-07-12T15:42:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-136"}]: dispatch 2023-07-12T15:42:06.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:05 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:07.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-136"}]': finished 2023-07-12T15:42:07.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[102777]: osdmap e671: 8 total, 8 up, 8 in 2023-07-12T15:42:07.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-136", "mode": "writeback"}]: dispatch 2023-07-12T15:42:07.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[102777]: pgmap v1022: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:07.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:42:07.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-136", "mode": "writeback"}]': finished 2023-07-12T15:42:07.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[102777]: osdmap e672: 8 total, 8 up, 8 in 2023-07-12T15:42:07.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-136"}]': finished 2023-07-12T15:42:07.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[94241]: osdmap e671: 8 total, 8 up, 8 in 2023-07-12T15:42:07.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-136", "mode": "writeback"}]: dispatch 2023-07-12T15:42:07.239 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[94241]: pgmap v1022: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:07.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:42:07.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-136", "mode": "writeback"}]': finished 2023-07-12T15:42:07.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:06 smithi069 ceph-mon[94241]: osdmap e672: 8 total, 8 up, 8 in 2023-07-12T15:42:07.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-136"}]': finished 2023-07-12T15:42:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:06 smithi145 ceph-mon[95801]: osdmap e671: 8 total, 8 up, 8 in 2023-07-12T15:42:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-136", "mode": "writeback"}]: dispatch 2023-07-12T15:42:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:06 smithi145 ceph-mon[95801]: pgmap v1022: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:06 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:42:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-136", "mode": "writeback"}]': finished 2023-07-12T15:42:07.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:06 smithi145 ceph-mon[95801]: osdmap e672: 8 total, 8 up, 8 in 2023-07-12T15:42:08.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:08.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:08.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:09.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:42:09.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:08 smithi069 ceph-mon[102777]: osdmap e673: 8 total, 8 up, 8 in 2023-07-12T15:42:09.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-136"}]: dispatch 2023-07-12T15:42:09.237 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:08 smithi069 ceph-mon[102777]: pgmap v1025: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:09.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:42:09.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:08 smithi069 ceph-mon[94241]: osdmap e673: 8 total, 8 up, 8 in 2023-07-12T15:42:09.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-136"}]: dispatch 2023-07-12T15:42:09.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:08 smithi069 ceph-mon[94241]: pgmap v1025: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:09.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:42:09.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:08 smithi145 ceph-mon[95801]: osdmap e673: 8 total, 8 up, 8 in 2023-07-12T15:42:09.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-136"}]: dispatch 2023-07-12T15:42:09.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:08 smithi145 ceph-mon[95801]: pgmap v1025: 204 pgs: 204 active+clean; 582 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:10.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:10 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:42:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-136"}]': finished 2023-07-12T15:42:10.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:10 smithi069 ceph-mon[102777]: osdmap e674: 8 total, 8 up, 8 in 2023-07-12T15:42:10.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:10 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:42:10.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-136"}]': finished 2023-07-12T15:42:10.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:10 smithi069 ceph-mon[94241]: osdmap e674: 8 total, 8 up, 8 in 2023-07-12T15:42:10.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:10 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:42:10.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-136"}]': finished 2023-07-12T15:42:10.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:10 smithi145 ceph-mon[95801]: osdmap e674: 8 total, 8 up, 8 in 2023-07-12T15:42:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:11 smithi069 ceph-mon[102777]: osdmap e675: 8 total, 8 up, 8 in 2023-07-12T15:42:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:11 smithi069 ceph-mon[102777]: pgmap v1028: 172 pgs: 172 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2023-07-12T15:42:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:11 smithi069 ceph-mon[102777]: osdmap e676: 8 total, 8 up, 8 in 2023-07-12T15:42:11.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:11.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:11 smithi069 ceph-mon[94241]: osdmap e675: 8 total, 8 up, 8 in 2023-07-12T15:42:11.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:11 smithi069 ceph-mon[94241]: pgmap v1028: 172 pgs: 172 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2023-07-12T15:42:11.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:11 smithi069 ceph-mon[94241]: osdmap e676: 8 total, 8 up, 8 in 2023-07-12T15:42:11.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:11.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:11 smithi145 ceph-mon[95801]: osdmap e675: 8 total, 8 up, 8 in 2023-07-12T15:42:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:11 smithi145 ceph-mon[95801]: pgmap v1028: 172 pgs: 172 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2023-07-12T15:42:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:11 smithi145 ceph-mon[95801]: osdmap e676: 8 total, 8 up, 8 in 2023-07-12T15:42:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:13.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-138","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:13.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:12 smithi069 ceph-mon[94241]: osdmap e677: 8 total, 8 up, 8 in 2023-07-12T15:42:13.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-138", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:13.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:12 smithi069 ceph-mon[94241]: pgmap v1031: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2023-07-12T15:42:13.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-138","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:13.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:12 smithi069 ceph-mon[102777]: osdmap e677: 8 total, 8 up, 8 in 2023-07-12T15:42:13.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-138", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:13.239 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:12 smithi069 ceph-mon[102777]: pgmap v1031: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2023-07-12T15:42:13.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-138","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:12 smithi145 ceph-mon[95801]: osdmap e677: 8 total, 8 up, 8 in 2023-07-12T15:42:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-138", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:12 smithi145 ceph-mon[95801]: pgmap v1031: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2023-07-12T15:42:14.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-138", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:14.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:13 smithi069 ceph-mon[102777]: osdmap e678: 8 total, 8 up, 8 in 2023-07-12T15:42:14.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:42:14.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:42:14.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-138", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:14.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:13 smithi069 ceph-mon[94241]: osdmap e678: 8 total, 8 up, 8 in 2023-07-12T15:42:14.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:42:14.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:42:14.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-138", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:13 smithi145 ceph-mon[95801]: osdmap e678: 8 total, 8 up, 8 in 2023-07-12T15:42:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:42:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:42:15.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:42:15 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:42:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:42:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:14 smithi145 ceph-mon[95801]: osdmap e679: 8 total, 8 up, 8 in 2023-07-12T15:42:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:42:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:14 smithi145 ceph-mon[95801]: pgmap v1034: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:15.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:42:15.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:14 smithi145 ceph-mon[95801]: osdmap e680: 8 total, 8 up, 8 in 2023-07-12T15:42:15.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:42:15.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:42:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[102777]: osdmap e679: 8 total, 8 up, 8 in 2023-07-12T15:42:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:42:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[102777]: pgmap v1034: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:42:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[102777]: osdmap e680: 8 total, 8 up, 8 in 2023-07-12T15:42:15.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:42:15.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:42:15.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[94241]: osdmap e679: 8 total, 8 up, 8 in 2023-07-12T15:42:15.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:42:15.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[94241]: pgmap v1034: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:15.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:42:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[94241]: osdmap e680: 8 total, 8 up, 8 in 2023-07-12T15:42:15.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2023-07-12T15:42:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:15 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_type","val": "explicit_object"}]': finished 2023-07-12T15:42:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:15 smithi145 ceph-mon[95801]: osdmap e681: 8 total, 8 up, 8 in 2023-07-12T15:42:16.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:15 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:16.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_type","val": "explicit_object"}]': finished 2023-07-12T15:42:16.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:15 smithi069 ceph-mon[102777]: osdmap e681: 8 total, 8 up, 8 in 2023-07-12T15:42:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:15 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-138","var": "hit_set_type","val": "explicit_object"}]': finished 2023-07-12T15:42:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:15 smithi069 ceph-mon[94241]: osdmap e681: 8 total, 8 up, 8 in 2023-07-12T15:42:16.487 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:42:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:42:16] "GET /metrics HTTP/1.1" 200 228555 "" "Prometheus/2.43.0" 2023-07-12T15:42:17.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:16 smithi145 ceph-mon[95801]: pgmap v1037: 204 pgs: 204 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:17.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:16 smithi069 ceph-mon[102777]: pgmap v1037: 204 pgs: 204 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:17.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:16 smithi069 ceph-mon[94241]: pgmap v1037: 204 pgs: 204 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:18.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:18.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-138"}]: dispatch 2023-07-12T15:42:18.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:18.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-138"}]: dispatch 2023-07-12T15:42:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:18.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-138"}]: dispatch 2023-07-12T15:42:19.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-138"}]': finished 2023-07-12T15:42:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:19 smithi145 ceph-mon[95801]: osdmap e682: 8 total, 8 up, 8 in 2023-07-12T15:42:19.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:19 smithi145 ceph-mon[95801]: pgmap v1039: 204 pgs: 204 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:19.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-138"}]': finished 2023-07-12T15:42:19.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:19 smithi069 ceph-mon[102777]: osdmap e682: 8 total, 8 up, 8 in 2023-07-12T15:42:19.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:19 smithi069 ceph-mon[102777]: pgmap v1039: 204 pgs: 204 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-138"}]': finished 2023-07-12T15:42:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:19 smithi069 ceph-mon[94241]: osdmap e682: 8 total, 8 up, 8 in 2023-07-12T15:42:19.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:19 smithi069 ceph-mon[94241]: pgmap v1039: 204 pgs: 204 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:20.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:20 smithi145 ceph-mon[95801]: osdmap e683: 8 total, 8 up, 8 in 2023-07-12T15:42:20.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:20 smithi069 ceph-mon[102777]: osdmap e683: 8 total, 8 up, 8 in 2023-07-12T15:42:20.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:20 smithi069 ceph-mon[94241]: osdmap e683: 8 total, 8 up, 8 in 2023-07-12T15:42:21.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:21 smithi145 ceph-mon[95801]: osdmap e684: 8 total, 8 up, 8 in 2023-07-12T15:42:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:21 smithi145 ceph-mon[95801]: pgmap v1042: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-140","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:21.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:21 smithi145 ceph-mon[95801]: osdmap e685: 8 total, 8 up, 8 in 2023-07-12T15:42:21.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-140", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:21.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:21 smithi069 ceph-mon[102777]: osdmap e684: 8 total, 8 up, 8 in 2023-07-12T15:42:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:21 smithi069 ceph-mon[102777]: pgmap v1042: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-140","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:21 smithi069 ceph-mon[102777]: osdmap e685: 8 total, 8 up, 8 in 2023-07-12T15:42:21.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-140", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:21.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:21 smithi069 ceph-mon[94241]: osdmap e684: 8 total, 8 up, 8 in 2023-07-12T15:42:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:21 smithi069 ceph-mon[94241]: pgmap v1042: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-140","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:21 smithi069 ceph-mon[94241]: osdmap e685: 8 total, 8 up, 8 in 2023-07-12T15:42:21.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-140", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:23.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-140", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:23 smithi145 ceph-mon[95801]: osdmap e686: 8 total, 8 up, 8 in 2023-07-12T15:42:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:23 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_count","val": "3"}]: dispatch 2023-07-12T15:42:23.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:23 smithi145 ceph-mon[95801]: pgmap v1045: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:23.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-140", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:23 smithi069 ceph-mon[94241]: osdmap e686: 8 total, 8 up, 8 in 2023-07-12T15:42:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:23 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_count","val": "3"}]: dispatch 2023-07-12T15:42:23.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:23 smithi069 ceph-mon[94241]: pgmap v1045: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:23.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-140", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:23.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:23 smithi069 ceph-mon[102777]: osdmap e686: 8 total, 8 up, 8 in 2023-07-12T15:42:23.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:23 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_count","val": "3"}]: dispatch 2023-07-12T15:42:23.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:23 smithi069 ceph-mon[102777]: pgmap v1045: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:24.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_count","val": "3"}]': finished 2023-07-12T15:42:24.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:24 smithi145 ceph-mon[95801]: osdmap e687: 8 total, 8 up, 8 in 2023-07-12T15:42:24.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:24 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_period","val": "3"}]: dispatch 2023-07-12T15:42:24.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_count","val": "3"}]': finished 2023-07-12T15:42:24.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:24 smithi069 ceph-mon[102777]: osdmap e687: 8 total, 8 up, 8 in 2023-07-12T15:42:24.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:24 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_period","val": "3"}]: dispatch 2023-07-12T15:42:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_count","val": "3"}]': finished 2023-07-12T15:42:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:24 smithi069 ceph-mon[94241]: osdmap e687: 8 total, 8 up, 8 in 2023-07-12T15:42:24.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:24 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_period","val": "3"}]: dispatch 2023-07-12T15:42:25.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:42:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:42:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_period","val": "3"}]': finished 2023-07-12T15:42:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:25 smithi145 ceph-mon[95801]: osdmap e688: 8 total, 8 up, 8 in 2023-07-12T15:42:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:42:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:25 smithi145 ceph-mon[95801]: pgmap v1048: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:25.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:25 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:42:25.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:25 smithi145 ceph-mon[95801]: osdmap e689: 8 total, 8 up, 8 in 2023-07-12T15:42:25.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_period","val": "3"}]': finished 2023-07-12T15:42:25.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:25 smithi069 ceph-mon[102777]: osdmap e688: 8 total, 8 up, 8 in 2023-07-12T15:42:25.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:42:25.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:25 smithi069 ceph-mon[102777]: pgmap v1048: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:25.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:25 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:42:25.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:25 smithi069 ceph-mon[102777]: osdmap e689: 8 total, 8 up, 8 in 2023-07-12T15:42:25.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_period","val": "3"}]': finished 2023-07-12T15:42:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:25 smithi069 ceph-mon[94241]: osdmap e688: 8 total, 8 up, 8 in 2023-07-12T15:42:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:42:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:25 smithi069 ceph-mon[94241]: pgmap v1048: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:25 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:42:25.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:25 smithi069 ceph-mon[94241]: osdmap e689: 8 total, 8 up, 8 in 2023-07-12T15:42:26.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2023-07-12T15:42:26.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:26 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:26.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:26 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_fpp","val": ".01"}]': finished 2023-07-12T15:42:26.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:26 smithi145 ceph-mon[95801]: osdmap e690: 8 total, 8 up, 8 in 2023-07-12T15:42:26.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2023-07-12T15:42:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:26 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:26 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_fpp","val": ".01"}]': finished 2023-07-12T15:42:26.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:26 smithi069 ceph-mon[102777]: osdmap e690: 8 total, 8 up, 8 in 2023-07-12T15:42:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2023-07-12T15:42:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:26 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:26 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-140","var": "hit_set_fpp","val": ".01"}]': finished 2023-07-12T15:42:26.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:26 smithi069 ceph-mon[94241]: osdmap e690: 8 total, 8 up, 8 in 2023-07-12T15:42:26.487 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:42:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:42:26] "GET /metrics HTTP/1.1" 200 228555 "" "Prometheus/2.43.0" 2023-07-12T15:42:27.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:27 smithi145 ceph-mon[95801]: pgmap v1051: 204 pgs: 204 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:27.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:27 smithi069 ceph-mon[94241]: pgmap v1051: 204 pgs: 204 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:27.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:27 smithi069 ceph-mon[102777]: pgmap v1051: 204 pgs: 204 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:29 smithi145 ceph-mon[95801]: pgmap v1052: 204 pgs: 204 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2023-07-12T15:42:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:42:29.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:29 smithi069 ceph-mon[102777]: pgmap v1052: 204 pgs: 204 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2023-07-12T15:42:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:42:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:29 smithi069 ceph-mon[94241]: pgmap v1052: 204 pgs: 204 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2023-07-12T15:42:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:42:31.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:31 smithi145 ceph-mon[95801]: pgmap v1053: 204 pgs: 204 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.3 KiB/s wr, 2 op/s 2023-07-12T15:42:31.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:31 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:42:31.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:31 smithi069 ceph-mon[102777]: pgmap v1053: 204 pgs: 204 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.3 KiB/s wr, 2 op/s 2023-07-12T15:42:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:31 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:42:31.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:31 smithi069 ceph-mon[94241]: pgmap v1053: 204 pgs: 204 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.3 KiB/s wr, 2 op/s 2023-07-12T15:42:31.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:31 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:42:32.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:32 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:42:32.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:32 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:42:32.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:32 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:42:32.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:32 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:42:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:32 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:42:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:32 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:42:32.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:32 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:42:32.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:32 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:42:32.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:32 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:42:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:33 smithi145 ceph-mon[95801]: pgmap v1054: 204 pgs: 204 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.1 KiB/s wr, 1 op/s 2023-07-12T15:42:33.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:33 smithi069 ceph-mon[102777]: pgmap v1054: 204 pgs: 204 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.1 KiB/s wr, 1 op/s 2023-07-12T15:42:33.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:33 smithi069 ceph-mon[94241]: pgmap v1054: 204 pgs: 204 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.1 KiB/s wr, 1 op/s 2023-07-12T15:42:35.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:42:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:42:35.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:35 smithi145 ceph-mon[95801]: pgmap v1055: 204 pgs: 204 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 446 B/s rd, 3.6 KiB/s wr, 0 op/s 2023-07-12T15:42:35.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:35 smithi069 ceph-mon[102777]: pgmap v1055: 204 pgs: 204 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 446 B/s rd, 3.6 KiB/s wr, 0 op/s 2023-07-12T15:42:35.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:35 smithi069 ceph-mon[94241]: pgmap v1055: 204 pgs: 204 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 446 B/s rd, 3.6 KiB/s wr, 0 op/s 2023-07-12T15:42:36.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:42:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:42:36] "GET /metrics HTTP/1.1" 200 228550 "" "Prometheus/2.43.0" 2023-07-12T15:42:36.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:36 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:36.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:36 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:36.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:36 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:37.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:37 smithi145 ceph-mon[95801]: pgmap v1056: 204 pgs: 204 active+clean; 4.5 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 7.3 KiB/s wr, 1 op/s 2023-07-12T15:42:37.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:37 smithi069 ceph-mon[102777]: pgmap v1056: 204 pgs: 204 active+clean; 4.5 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 7.3 KiB/s wr, 1 op/s 2023-07-12T15:42:37.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:37 smithi069 ceph-mon[94241]: pgmap v1056: 204 pgs: 204 active+clean; 4.5 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 7.3 KiB/s wr, 1 op/s 2023-07-12T15:42:38.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:38.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-140"}]: dispatch 2023-07-12T15:42:38.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:38.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-140"}]: dispatch 2023-07-12T15:42:38.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:38.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-140"}]: dispatch 2023-07-12T15:42:39.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:39 smithi145 ceph-mon[95801]: pgmap v1057: 204 pgs: 204 active+clean; 4.5 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 6.2 KiB/s wr, 1 op/s 2023-07-12T15:42:39.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:39 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-140"}]': finished 2023-07-12T15:42:39.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:39 smithi145 ceph-mon[95801]: osdmap e691: 8 total, 8 up, 8 in 2023-07-12T15:42:39.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:39 smithi069 ceph-mon[102777]: pgmap v1057: 204 pgs: 204 active+clean; 4.5 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 6.2 KiB/s wr, 1 op/s 2023-07-12T15:42:39.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:39 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-140"}]': finished 2023-07-12T15:42:39.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:39 smithi069 ceph-mon[102777]: osdmap e691: 8 total, 8 up, 8 in 2023-07-12T15:42:39.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:39 smithi069 ceph-mon[94241]: pgmap v1057: 204 pgs: 204 active+clean; 4.5 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 6.2 KiB/s wr, 1 op/s 2023-07-12T15:42:39.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:39 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-140"}]': finished 2023-07-12T15:42:39.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:39 smithi069 ceph-mon[94241]: osdmap e691: 8 total, 8 up, 8 in 2023-07-12T15:42:40.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:40 smithi145 ceph-mon[95801]: osdmap e692: 8 total, 8 up, 8 in 2023-07-12T15:42:40.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:40 smithi145 ceph-mon[95801]: osdmap e693: 8 total, 8 up, 8 in 2023-07-12T15:42:40.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:40.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:40 smithi069 ceph-mon[102777]: osdmap e692: 8 total, 8 up, 8 in 2023-07-12T15:42:40.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:40 smithi069 ceph-mon[102777]: osdmap e693: 8 total, 8 up, 8 in 2023-07-12T15:42:40.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:40.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:40 smithi069 ceph-mon[94241]: osdmap e692: 8 total, 8 up, 8 in 2023-07-12T15:42:40.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:40 smithi069 ceph-mon[94241]: osdmap e693: 8 total, 8 up, 8 in 2023-07-12T15:42:40.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:41.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:41 smithi145 ceph-mon[95801]: pgmap v1060: 172 pgs: 172 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-142","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:41 smithi145 ceph-mon[95801]: osdmap e694: 8 total, 8 up, 8 in 2023-07-12T15:42:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:41.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:41 smithi069 ceph-mon[102777]: pgmap v1060: 172 pgs: 172 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:41.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-142","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:41.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:41 smithi069 ceph-mon[102777]: osdmap e694: 8 total, 8 up, 8 in 2023-07-12T15:42:41.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:41.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:41 smithi069 ceph-mon[94241]: pgmap v1060: 172 pgs: 172 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:41.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-142","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:41.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:41 smithi069 ceph-mon[94241]: osdmap e694: 8 total, 8 up, 8 in 2023-07-12T15:42:41.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:43 smithi145 ceph-mon[95801]: pgmap v1063: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:43 smithi145 ceph-mon[95801]: osdmap e695: 8 total, 8 up, 8 in 2023-07-12T15:42:43.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-142"}]: dispatch 2023-07-12T15:42:43.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:43 smithi069 ceph-mon[102777]: pgmap v1063: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:43.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:43.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:43 smithi069 ceph-mon[102777]: osdmap e695: 8 total, 8 up, 8 in 2023-07-12T15:42:43.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-142"}]: dispatch 2023-07-12T15:42:43.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:43 smithi069 ceph-mon[94241]: pgmap v1063: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:43.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:43.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:43 smithi069 ceph-mon[94241]: osdmap e695: 8 total, 8 up, 8 in 2023-07-12T15:42:43.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-142"}]: dispatch 2023-07-12T15:42:44.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-142"}]': finished 2023-07-12T15:42:44.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:44 smithi145 ceph-mon[95801]: osdmap e696: 8 total, 8 up, 8 in 2023-07-12T15:42:44.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-142", "mode": "writeback"}]: dispatch 2023-07-12T15:42:44.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:44 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:42:44.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-142"}]': finished 2023-07-12T15:42:44.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:44 smithi069 ceph-mon[102777]: osdmap e696: 8 total, 8 up, 8 in 2023-07-12T15:42:44.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-142", "mode": "writeback"}]: dispatch 2023-07-12T15:42:44.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:44 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:42:44.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-142"}]': finished 2023-07-12T15:42:44.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:44 smithi069 ceph-mon[94241]: osdmap e696: 8 total, 8 up, 8 in 2023-07-12T15:42:44.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-142", "mode": "writeback"}]: dispatch 2023-07-12T15:42:44.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:44 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:42:45.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:42:45 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:42:45.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:45 smithi145 ceph-mon[95801]: pgmap v1066: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:45.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:45 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:42:45.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-142", "mode": "writeback"}]': finished 2023-07-12T15:42:45.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:45 smithi145 ceph-mon[95801]: osdmap e697: 8 total, 8 up, 8 in 2023-07-12T15:42:45.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:42:45.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:45 smithi069 ceph-mon[94241]: pgmap v1066: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:45.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:45 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:42:45.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-142", "mode": "writeback"}]': finished 2023-07-12T15:42:45.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:45 smithi069 ceph-mon[94241]: osdmap e697: 8 total, 8 up, 8 in 2023-07-12T15:42:45.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:42:45.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:45 smithi069 ceph-mon[102777]: pgmap v1066: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:45.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:45 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:42:45.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-142", "mode": "writeback"}]': finished 2023-07-12T15:42:45.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:45 smithi069 ceph-mon[102777]: osdmap e697: 8 total, 8 up, 8 in 2023-07-12T15:42:45.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:42:46.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:42:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:42:46] "GET /metrics HTTP/1.1" 200 228503 "" "Prometheus/2.43.0" 2023-07-12T15:42:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:46 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:42:46.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:46 smithi069 ceph-mon[94241]: osdmap e698: 8 total, 8 up, 8 in 2023-07-12T15:42:46.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:46 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:42:46.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:46 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:46.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:42:46.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:46 smithi069 ceph-mon[102777]: osdmap e698: 8 total, 8 up, 8 in 2023-07-12T15:42:46.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:46 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:42:46.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:46 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:46.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:42:46.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:46 smithi145 ceph-mon[95801]: osdmap e698: 8 total, 8 up, 8 in 2023-07-12T15:42:46.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:46 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:42:47.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:47 smithi145 ceph-mon[95801]: pgmap v1069: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-07-12T15:42:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:42:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:47 smithi145 ceph-mon[95801]: osdmap e699: 8 total, 8 up, 8 in 2023-07-12T15:42:47.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:42:47.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:47 smithi069 ceph-mon[94241]: pgmap v1069: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-07-12T15:42:47.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:42:47.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:47 smithi069 ceph-mon[94241]: osdmap e699: 8 total, 8 up, 8 in 2023-07-12T15:42:47.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:42:47.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:47 smithi069 ceph-mon[102777]: pgmap v1069: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-07-12T15:42:47.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:42:47.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:47 smithi069 ceph-mon[102777]: osdmap e699: 8 total, 8 up, 8 in 2023-07-12T15:42:47.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:42:48.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:48 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:42:48.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:42:48.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:48 smithi145 ceph-mon[95801]: osdmap e700: 8 total, 8 up, 8 in 2023-07-12T15:42:48.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2023-07-12T15:42:48.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2023-07-12T15:42:48.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:48 smithi145 ceph-mon[95801]: osdmap e701: 8 total, 8 up, 8 in 2023-07-12T15:42:48.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:48 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:42:48.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:42:48.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:48 smithi069 ceph-mon[94241]: osdmap e700: 8 total, 8 up, 8 in 2023-07-12T15:42:48.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2023-07-12T15:42:48.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2023-07-12T15:42:48.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:48 smithi069 ceph-mon[94241]: osdmap e701: 8 total, 8 up, 8 in 2023-07-12T15:42:48.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:48 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:42:48.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:42:48.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:48 smithi069 ceph-mon[102777]: osdmap e700: 8 total, 8 up, 8 in 2023-07-12T15:42:48.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2023-07-12T15:42:48.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2023-07-12T15:42:48.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:48 smithi069 ceph-mon[102777]: osdmap e701: 8 total, 8 up, 8 in 2023-07-12T15:42:49.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:49 smithi145 ceph-mon[95801]: pgmap v1072: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-07-12T15:42:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2023-07-12T15:42:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2023-07-12T15:42:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:49 smithi145 ceph-mon[95801]: osdmap e702: 8 total, 8 up, 8 in 2023-07-12T15:42:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2023-07-12T15:42:49.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:49 smithi069 ceph-mon[102777]: pgmap v1072: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-07-12T15:42:49.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2023-07-12T15:42:49.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2023-07-12T15:42:49.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:49 smithi069 ceph-mon[102777]: osdmap e702: 8 total, 8 up, 8 in 2023-07-12T15:42:49.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2023-07-12T15:42:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:49 smithi069 ceph-mon[94241]: pgmap v1072: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-07-12T15:42:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2023-07-12T15:42:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2023-07-12T15:42:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:49 smithi069 ceph-mon[94241]: osdmap e702: 8 total, 8 up, 8 in 2023-07-12T15:42:49.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2023-07-12T15:42:51.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:51 smithi145 ceph-mon[95801]: pgmap v1075: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:51.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_search_last_n","val": "1"}]': finished 2023-07-12T15:42:51.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:51 smithi145 ceph-mon[95801]: osdmap e703: 8 total, 8 up, 8 in 2023-07-12T15:42:51.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:51.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:51 smithi069 ceph-mon[102777]: pgmap v1075: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:51.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_search_last_n","val": "1"}]': finished 2023-07-12T15:42:51.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:51 smithi069 ceph-mon[102777]: osdmap e703: 8 total, 8 up, 8 in 2023-07-12T15:42:51.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:51.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:51 smithi069 ceph-mon[94241]: pgmap v1075: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:51.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-142","var": "hit_set_search_last_n","val": "1"}]': finished 2023-07-12T15:42:51.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:51 smithi069 ceph-mon[94241]: osdmap e703: 8 total, 8 up, 8 in 2023-07-12T15:42:51.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:52.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:42:52.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:52 smithi069 ceph-mon[102777]: osdmap e704: 8 total, 8 up, 8 in 2023-07-12T15:42:52.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142"}]: dispatch 2023-07-12T15:42:52.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:42:52.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:52 smithi069 ceph-mon[94241]: osdmap e704: 8 total, 8 up, 8 in 2023-07-12T15:42:52.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142"}]: dispatch 2023-07-12T15:42:52.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:42:52.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:52 smithi145 ceph-mon[95801]: osdmap e704: 8 total, 8 up, 8 in 2023-07-12T15:42:52.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142"}]: dispatch 2023-07-12T15:42:53.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:53 smithi069 ceph-mon[102777]: pgmap v1078: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:53.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142"}]': finished 2023-07-12T15:42:53.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:53 smithi069 ceph-mon[102777]: osdmap e705: 8 total, 8 up, 8 in 2023-07-12T15:42:53.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:53.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142"}]: dispatch 2023-07-12T15:42:53.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:53 smithi069 ceph-mon[94241]: pgmap v1078: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:53.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142"}]': finished 2023-07-12T15:42:53.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:53 smithi069 ceph-mon[94241]: osdmap e705: 8 total, 8 up, 8 in 2023-07-12T15:42:53.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:53.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142"}]: dispatch 2023-07-12T15:42:53.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:53 smithi145 ceph-mon[95801]: pgmap v1078: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:42:53.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142"}]': finished 2023-07-12T15:42:53.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:53 smithi145 ceph-mon[95801]: osdmap e705: 8 total, 8 up, 8 in 2023-07-12T15:42:53.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:42:53.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-142"}]: dispatch 2023-07-12T15:42:54.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:54 smithi069 ceph-mon[102777]: osdmap e706: 8 total, 8 up, 8 in 2023-07-12T15:42:54.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:54 smithi069 ceph-mon[94241]: osdmap e706: 8 total, 8 up, 8 in 2023-07-12T15:42:54.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:54 smithi145 ceph-mon[95801]: osdmap e706: 8 total, 8 up, 8 in 2023-07-12T15:42:55.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:42:55 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:42:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:55 smithi145 ceph-mon[95801]: pgmap v1081: 172 pgs: 172 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:55 smithi145 ceph-mon[95801]: osdmap e707: 8 total, 8 up, 8 in 2023-07-12T15:42:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:55.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:55 smithi069 ceph-mon[102777]: pgmap v1081: 172 pgs: 172 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:55.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:55 smithi069 ceph-mon[102777]: osdmap e707: 8 total, 8 up, 8 in 2023-07-12T15:42:55.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:55.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:55 smithi069 ceph-mon[94241]: pgmap v1081: 172 pgs: 172 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:42:55.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:55 smithi069 ceph-mon[94241]: osdmap e707: 8 total, 8 up, 8 in 2023-07-12T15:42:55.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:42:56.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:42:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:42:56] "GET /metrics HTTP/1.1" 200 228503 "" "Prometheus/2.43.0" 2023-07-12T15:42:56.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:56 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:56.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-144","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:56.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:56 smithi069 ceph-mon[94241]: osdmap e708: 8 total, 8 up, 8 in 2023-07-12T15:42:56.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:56 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:56 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-144","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:56 smithi069 ceph-mon[102777]: osdmap e708: 8 total, 8 up, 8 in 2023-07-12T15:42:56.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:56 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:56.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:56 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:42:56.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-144","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:42:56.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:56 smithi145 ceph-mon[95801]: osdmap e708: 8 total, 8 up, 8 in 2023-07-12T15:42:56.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:56 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:42:57.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:57 smithi069 ceph-mon[102777]: pgmap v1084: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-07-12T15:42:57.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:57.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:57 smithi069 ceph-mon[102777]: osdmap e709: 8 total, 8 up, 8 in 2023-07-12T15:42:57.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:57 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-144"}]: dispatch 2023-07-12T15:42:57.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:57 smithi069 ceph-mon[94241]: pgmap v1084: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-07-12T15:42:57.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:57.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:57 smithi069 ceph-mon[94241]: osdmap e709: 8 total, 8 up, 8 in 2023-07-12T15:42:57.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:57 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-144"}]: dispatch 2023-07-12T15:42:57.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:57 smithi145 ceph-mon[95801]: pgmap v1084: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-07-12T15:42:57.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:42:57.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:57 smithi145 ceph-mon[95801]: osdmap e709: 8 total, 8 up, 8 in 2023-07-12T15:42:57.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:57 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-144"}]: dispatch 2023-07-12T15:42:58.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-144"}]': finished 2023-07-12T15:42:58.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:58 smithi069 ceph-mon[102777]: osdmap e710: 8 total, 8 up, 8 in 2023-07-12T15:42:58.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:58 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-144", "mode": "readproxy"}]: dispatch 2023-07-12T15:42:58.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:58 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:42:58.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-144"}]': finished 2023-07-12T15:42:58.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:58 smithi069 ceph-mon[94241]: osdmap e710: 8 total, 8 up, 8 in 2023-07-12T15:42:58.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:58 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-144", "mode": "readproxy"}]: dispatch 2023-07-12T15:42:58.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:58 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:42:58.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-144"}]': finished 2023-07-12T15:42:58.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:58 smithi145 ceph-mon[95801]: osdmap e710: 8 total, 8 up, 8 in 2023-07-12T15:42:58.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:58 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-144", "mode": "readproxy"}]: dispatch 2023-07-12T15:42:58.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:58 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:42:59.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:59 smithi069 ceph-mon[94241]: pgmap v1087: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-07-12T15:42:59.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:59 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-144", "mode": "readproxy"}]': finished 2023-07-12T15:42:59.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:59 smithi069 ceph-mon[94241]: osdmap e711: 8 total, 8 up, 8 in 2023-07-12T15:42:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:59 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:42:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:42:59 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:42:59.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:59 smithi069 ceph-mon[102777]: pgmap v1087: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-07-12T15:42:59.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:59 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-144", "mode": "readproxy"}]': finished 2023-07-12T15:42:59.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:59 smithi069 ceph-mon[102777]: osdmap e711: 8 total, 8 up, 8 in 2023-07-12T15:42:59.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:59 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:42:59.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:42:59 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:42:59.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:59 smithi145 ceph-mon[95801]: pgmap v1087: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-07-12T15:42:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:59 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-144", "mode": "readproxy"}]': finished 2023-07-12T15:42:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:59 smithi145 ceph-mon[95801]: osdmap e711: 8 total, 8 up, 8 in 2023-07-12T15:42:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:59 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:42:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:42:59 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:43:01.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:01 smithi069 ceph-mon[102777]: pgmap v1089: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 211 B/s wr, 1 op/s 2023-07-12T15:43:01.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:01 smithi069 ceph-mon[94241]: pgmap v1089: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 211 B/s wr, 1 op/s 2023-07-12T15:43:01.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:01 smithi145 ceph-mon[95801]: pgmap v1089: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 211 B/s wr, 1 op/s 2023-07-12T15:43:03.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:02 smithi069 ceph-mon[102777]: pgmap v1090: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-07-12T15:43:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:02 smithi069 ceph-mon[94241]: pgmap v1090: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-07-12T15:43:03.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:02 smithi145 ceph-mon[95801]: pgmap v1090: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2023-07-12T15:43:05.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:05 smithi145 ceph-mon[95801]: pgmap v1091: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 130 B/s wr, 0 op/s 2023-07-12T15:43:05.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:43:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:43:05.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:05 smithi069 ceph-mon[102777]: pgmap v1091: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 130 B/s wr, 0 op/s 2023-07-12T15:43:05.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:05 smithi069 ceph-mon[94241]: pgmap v1091: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 130 B/s wr, 0 op/s 2023-07-12T15:43:06.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:06 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:06.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:06 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:06.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:43:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:43:06] "GET /metrics HTTP/1.1" 200 228527 "" "Prometheus/2.43.0" 2023-07-12T15:43:06.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:06 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:07.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:07 smithi145 ceph-mon[95801]: pgmap v1092: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 B/s wr, 1 op/s 2023-07-12T15:43:07.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:07 smithi069 ceph-mon[102777]: pgmap v1092: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 B/s wr, 1 op/s 2023-07-12T15:43:07.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:07 smithi069 ceph-mon[94241]: pgmap v1092: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 B/s wr, 1 op/s 2023-07-12T15:43:09.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:09 smithi145 ceph-mon[95801]: pgmap v1093: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2023-07-12T15:43:09.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:09.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:09 smithi069 ceph-mon[102777]: pgmap v1093: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2023-07-12T15:43:09.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:09.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:09 smithi069 ceph-mon[94241]: pgmap v1093: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2023-07-12T15:43:09.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:10.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:43:10.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:10 smithi145 ceph-mon[95801]: osdmap e712: 8 total, 8 up, 8 in 2023-07-12T15:43:10.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144"}]: dispatch 2023-07-12T15:43:10.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:43:10.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:10 smithi069 ceph-mon[102777]: osdmap e712: 8 total, 8 up, 8 in 2023-07-12T15:43:10.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144"}]: dispatch 2023-07-12T15:43:10.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:43:10.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:10 smithi069 ceph-mon[94241]: osdmap e712: 8 total, 8 up, 8 in 2023-07-12T15:43:10.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144"}]: dispatch 2023-07-12T15:43:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:11 smithi145 ceph-mon[95801]: pgmap v1095: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:43:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:11 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:43:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144"}]': finished 2023-07-12T15:43:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:11 smithi145 ceph-mon[95801]: osdmap e713: 8 total, 8 up, 8 in 2023-07-12T15:43:11.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:11.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144"}]: dispatch 2023-07-12T15:43:11.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:11 smithi069 ceph-mon[102777]: pgmap v1095: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:43:11.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:11 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:43:11.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144"}]': finished 2023-07-12T15:43:11.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:11 smithi069 ceph-mon[102777]: osdmap e713: 8 total, 8 up, 8 in 2023-07-12T15:43:11.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:11.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144"}]: dispatch 2023-07-12T15:43:11.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:11 smithi069 ceph-mon[94241]: pgmap v1095: 204 pgs: 204 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:43:11.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:11 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:43:11.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144"}]': finished 2023-07-12T15:43:11.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:11 smithi069 ceph-mon[94241]: osdmap e713: 8 total, 8 up, 8 in 2023-07-12T15:43:11.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:11.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-144"}]: dispatch 2023-07-12T15:43:12.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:12 smithi145 ceph-mon[95801]: osdmap e714: 8 total, 8 up, 8 in 2023-07-12T15:43:12.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:12 smithi069 ceph-mon[102777]: osdmap e714: 8 total, 8 up, 8 in 2023-07-12T15:43:12.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:12 smithi069 ceph-mon[94241]: osdmap e714: 8 total, 8 up, 8 in 2023-07-12T15:43:13.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:13 smithi145 ceph-mon[95801]: pgmap v1098: 172 pgs: 172 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:43:13.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:13 smithi145 ceph-mon[95801]: osdmap e715: 8 total, 8 up, 8 in 2023-07-12T15:43:13.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:13.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:13 smithi069 ceph-mon[102777]: pgmap v1098: 172 pgs: 172 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:43:13.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:13 smithi069 ceph-mon[102777]: osdmap e715: 8 total, 8 up, 8 in 2023-07-12T15:43:13.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:13.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:13 smithi069 ceph-mon[94241]: pgmap v1098: 172 pgs: 172 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:43:13.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:13 smithi069 ceph-mon[94241]: osdmap e715: 8 total, 8 up, 8 in 2023-07-12T15:43:13.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:14.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-146","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:14.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:14 smithi145 ceph-mon[95801]: osdmap e716: 8 total, 8 up, 8 in 2023-07-12T15:43:14.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:43:14.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:43:14.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:14 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:43:14.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-146","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:14.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:14 smithi069 ceph-mon[94241]: osdmap e716: 8 total, 8 up, 8 in 2023-07-12T15:43:14.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:43:14.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:43:14.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:14 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:43:14.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-146","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:14.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:14 smithi069 ceph-mon[102777]: osdmap e716: 8 total, 8 up, 8 in 2023-07-12T15:43:14.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:43:14.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:43:14.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:14 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:43:15.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:43:15 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:43:15.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:15 smithi145 ceph-mon[95801]: pgmap v1101: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:15.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:43:15.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:15 smithi145 ceph-mon[95801]: osdmap e717: 8 total, 8 up, 8 in 2023-07-12T15:43:15.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-146"}]: dispatch 2023-07-12T15:43:15.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:15 smithi069 ceph-mon[102777]: pgmap v1101: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:15.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:43:15.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:15 smithi069 ceph-mon[102777]: osdmap e717: 8 total, 8 up, 8 in 2023-07-12T15:43:15.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-146"}]: dispatch 2023-07-12T15:43:15.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:15 smithi069 ceph-mon[94241]: pgmap v1101: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 701 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:15.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:43:15.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:15 smithi069 ceph-mon[94241]: osdmap e717: 8 total, 8 up, 8 in 2023-07-12T15:43:15.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-146"}]: dispatch 2023-07-12T15:43:16.487 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:43:16] "GET /metrics HTTP/1.1" 200 228473 "" "Prometheus/2.43.0" 2023-07-12T15:43:16.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:16.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-146"}]': finished 2023-07-12T15:43:16.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[102777]: osdmap e718: 8 total, 8 up, 8 in 2023-07-12T15:43:16.488 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-146", "mode": "writeback"}]: dispatch 2023-07-12T15:43:16.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:43:16.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-146", "mode": "writeback"}]': finished 2023-07-12T15:43:16.489 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[102777]: osdmap e719: 8 total, 8 up, 8 in 2023-07-12T15:43:16.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:16.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-146"}]': finished 2023-07-12T15:43:16.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[94241]: osdmap e718: 8 total, 8 up, 8 in 2023-07-12T15:43:16.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-146", "mode": "writeback"}]: dispatch 2023-07-12T15:43:16.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:43:16.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-146", "mode": "writeback"}]': finished 2023-07-12T15:43:16.491 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:16 smithi069 ceph-mon[94241]: osdmap e719: 8 total, 8 up, 8 in 2023-07-12T15:43:16.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:16 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:16.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi069-133790-111", "overlaypool": "test-rados-api-smithi069-133790-146"}]': finished 2023-07-12T15:43:16.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:16 smithi145 ceph-mon[95801]: osdmap e718: 8 total, 8 up, 8 in 2023-07-12T15:43:16.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-146", "mode": "writeback"}]: dispatch 2023-07-12T15:43:16.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:16 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-07-12T15:43:16.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:16 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi069-133790-146", "mode": "writeback"}]': finished 2023-07-12T15:43:16.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:16 smithi145 ceph-mon[95801]: osdmap e719: 8 total, 8 up, 8 in 2023-07-12T15:43:17.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:17 smithi069 ceph-mon[94241]: pgmap v1104: 204 pgs: 204 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:43:17.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:17 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:43:17.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:17 smithi069 ceph-mon[102777]: pgmap v1104: 204 pgs: 204 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:43:17.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:17 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:43:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:17 smithi145 ceph-mon[95801]: pgmap v1104: 204 pgs: 204 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:43:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:17 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_count","val": "2"}]: dispatch 2023-07-12T15:43:18.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:43:18.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:18 smithi069 ceph-mon[102777]: osdmap e720: 8 total, 8 up, 8 in 2023-07-12T15:43:18.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:18 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:43:18.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:43:18.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:18 smithi069 ceph-mon[94241]: osdmap e720: 8 total, 8 up, 8 in 2023-07-12T15:43:18.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:18 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:43:18.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_count","val": "2"}]': finished 2023-07-12T15:43:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:18 smithi145 ceph-mon[95801]: osdmap e720: 8 total, 8 up, 8 in 2023-07-12T15:43:18.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:18 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_period","val": "600"}]: dispatch 2023-07-12T15:43:19.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:19 smithi069 ceph-mon[102777]: pgmap v1107: 204 pgs: 204 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:43:19.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:43:19.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:19 smithi069 ceph-mon[102777]: osdmap e721: 8 total, 8 up, 8 in 2023-07-12T15:43:19.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:19 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:43:19.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:19 smithi069 ceph-mon[94241]: pgmap v1107: 204 pgs: 204 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:43:19.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:43:19.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:19 smithi069 ceph-mon[94241]: osdmap e721: 8 total, 8 up, 8 in 2023-07-12T15:43:19.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:19 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:43:19.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:19 smithi145 ceph-mon[95801]: pgmap v1107: 204 pgs: 204 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2023-07-12T15:43:19.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_period","val": "600"}]': finished 2023-07-12T15:43:19.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:19 smithi145 ceph-mon[95801]: osdmap e721: 8 total, 8 up, 8 in 2023-07-12T15:43:19.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:19 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_type","val": "bloom"}]: dispatch 2023-07-12T15:43:20.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:20 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:43:20.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:43:20.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:20 smithi069 ceph-mon[94241]: osdmap e722: 8 total, 8 up, 8 in 2023-07-12T15:43:20.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:20 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2023-07-12T15:43:20.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:20 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:43:20.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:43:20.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:20 smithi069 ceph-mon[102777]: osdmap e722: 8 total, 8 up, 8 in 2023-07-12T15:43:20.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:20 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2023-07-12T15:43:20.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:20 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-07-12T15:43:20.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "hit_set_type","val": "bloom"}]': finished 2023-07-12T15:43:20.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:20 smithi145 ceph-mon[95801]: osdmap e722: 8 total, 8 up, 8 in 2023-07-12T15:43:20.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:20 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2023-07-12T15:43:21.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:21 smithi069 ceph-mon[102777]: pgmap v1110: 204 pgs: 204 active+clean; 4.4 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2023-07-12T15:43:21.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2023-07-12T15:43:21.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:21 smithi069 ceph-mon[102777]: osdmap e723: 8 total, 8 up, 8 in 2023-07-12T15:43:21.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "target_max_objects","val": "1"}]: dispatch 2023-07-12T15:43:21.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:21 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "target_max_objects","val": "1"}]': finished 2023-07-12T15:43:21.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:21 smithi069 ceph-mon[102777]: osdmap e724: 8 total, 8 up, 8 in 2023-07-12T15:43:21.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:21 smithi069 ceph-mon[94241]: pgmap v1110: 204 pgs: 204 active+clean; 4.4 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2023-07-12T15:43:21.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2023-07-12T15:43:21.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:21 smithi069 ceph-mon[94241]: osdmap e723: 8 total, 8 up, 8 in 2023-07-12T15:43:21.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "target_max_objects","val": "1"}]: dispatch 2023-07-12T15:43:21.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:21 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "target_max_objects","val": "1"}]': finished 2023-07-12T15:43:21.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:21 smithi069 ceph-mon[94241]: osdmap e724: 8 total, 8 up, 8 in 2023-07-12T15:43:21.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:21 smithi145 ceph-mon[95801]: pgmap v1110: 204 pgs: 204 active+clean; 4.4 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2023-07-12T15:43:21.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2023-07-12T15:43:21.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:21 smithi145 ceph-mon[95801]: osdmap e723: 8 total, 8 up, 8 in 2023-07-12T15:43:21.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "target_max_objects","val": "1"}]: dispatch 2023-07-12T15:43:21.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:21 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-146","var": "target_max_objects","val": "1"}]': finished 2023-07-12T15:43:21.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:21 smithi145 ceph-mon[95801]: osdmap e724: 8 total, 8 up, 8 in 2023-07-12T15:43:23.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:22 smithi069 ceph-mon[102777]: pgmap v1113: 204 pgs: 204 active+clean; 4.4 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2023-07-12T15:43:23.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:22 smithi069 ceph-mon[94241]: pgmap v1113: 204 pgs: 204 active+clean; 4.4 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2023-07-12T15:43:23.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:22 smithi145 ceph-mon[95801]: pgmap v1113: 204 pgs: 204 active+clean; 4.4 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2023-07-12T15:43:24.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:23 smithi069 ceph-mon[102777]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2023-07-12T15:43:24.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:23 smithi069 ceph-mon[94241]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2023-07-12T15:43:24.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:23 smithi145 ceph-mon[95801]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2023-07-12T15:43:25.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:24 smithi069 ceph-mon[102777]: pgmap v1114: 204 pgs: 204 active+clean; 4.4 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2023-07-12T15:43:25.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:24 smithi069 ceph-mon[94241]: pgmap v1114: 204 pgs: 204 active+clean; 4.4 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2023-07-12T15:43:25.251 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:24 smithi145 ceph-mon[95801]: pgmap v1114: 204 pgs: 204 active+clean; 4.4 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2023-07-12T15:43:25.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:43:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:43:26.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:25 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:26.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:25 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:26.236 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:43:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:43:26] "GET /metrics HTTP/1.1" 200 228473 "" "Prometheus/2.43.0" 2023-07-12T15:43:26.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:25 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:27.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:26 smithi069 ceph-mon[94241]: pgmap v1115: 204 pgs: 204 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s 2023-07-12T15:43:27.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:26 smithi069 ceph-mon[102777]: pgmap v1115: 204 pgs: 204 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s 2023-07-12T15:43:27.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:26 smithi145 ceph-mon[95801]: pgmap v1115: 204 pgs: 204 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s 2023-07-12T15:43:29.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:29 smithi069 ceph-mon[102777]: pgmap v1116: 204 pgs: 204 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2023-07-12T15:43:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:29 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:43:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:43:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:29 smithi069 ceph-mon[94241]: pgmap v1116: 204 pgs: 204 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2023-07-12T15:43:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:29 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:43:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:43:29.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:29 smithi145 ceph-mon[95801]: pgmap v1116: 204 pgs: 204 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2023-07-12T15:43:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:29 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:43:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:43:31.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:31 smithi069 ceph-mon[102777]: pgmap v1117: 204 pgs: 204 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:43:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:31 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:31.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:31 smithi069 ceph-mon[94241]: pgmap v1117: 204 pgs: 204 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:43:31.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:31 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:31.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:31 smithi145 ceph-mon[95801]: pgmap v1117: 204 pgs: 204 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:43:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:31 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:43:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[102777]: osdmap e725: 8 total, 8 up, 8 in 2023-07-12T15:43:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146"}]: dispatch 2023-07-12T15:43:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:43:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:43:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:43:32.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:43:32.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146"}]': finished 2023-07-12T15:43:32.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[102777]: osdmap e726: 8 total, 8 up, 8 in 2023-07-12T15:43:32.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:43:32.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[94241]: osdmap e725: 8 total, 8 up, 8 in 2023-07-12T15:43:32.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146"}]: dispatch 2023-07-12T15:43:32.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:43:32.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:43:32.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:43:32.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:43:32.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146"}]': finished 2023-07-12T15:43:32.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:32 smithi069 ceph-mon[94241]: osdmap e726: 8 total, 8 up, 8 in 2023-07-12T15:43:32.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:43:32.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:32 smithi145 ceph-mon[95801]: osdmap e725: 8 total, 8 up, 8 in 2023-07-12T15:43:32.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146"}]: dispatch 2023-07-12T15:43:32.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:32 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:43:32.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:32 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:43:32.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:32 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:43:32.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:32 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:43:32.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146"}]': finished 2023-07-12T15:43:32.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:32 smithi145 ceph-mon[95801]: osdmap e726: 8 total, 8 up, 8 in 2023-07-12T15:43:33.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:33 smithi145 ceph-mon[95801]: pgmap v1119: 204 pgs: 204 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:43:33.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:33.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146"}]: dispatch 2023-07-12T15:43:33.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:33 smithi069 ceph-mon[102777]: pgmap v1119: 204 pgs: 204 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:43:33.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:33.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146"}]: dispatch 2023-07-12T15:43:33.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:33 smithi069 ceph-mon[94241]: pgmap v1119: 204 pgs: 204 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2023-07-12T15:43:33.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:33.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-146"}]: dispatch 2023-07-12T15:43:34.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:34 smithi145 ceph-mon[95801]: osdmap e727: 8 total, 8 up, 8 in 2023-07-12T15:43:34.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:34 smithi069 ceph-mon[102777]: osdmap e727: 8 total, 8 up, 8 in 2023-07-12T15:43:34.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:34 smithi069 ceph-mon[94241]: osdmap e727: 8 total, 8 up, 8 in 2023-07-12T15:43:35.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:43:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:43:35.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:35 smithi145 ceph-mon[95801]: pgmap v1122: 172 pgs: 172 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-12T15:43:35.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:35 smithi145 ceph-mon[95801]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2023-07-12T15:43:35.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:35 smithi145 ceph-mon[95801]: osdmap e728: 8 total, 8 up, 8 in 2023-07-12T15:43:35.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:35 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:35.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:35 smithi069 ceph-mon[102777]: pgmap v1122: 172 pgs: 172 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-12T15:43:35.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:35 smithi069 ceph-mon[102777]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2023-07-12T15:43:35.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:35 smithi069 ceph-mon[102777]: osdmap e728: 8 total, 8 up, 8 in 2023-07-12T15:43:35.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:35 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:35.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:35 smithi069 ceph-mon[94241]: pgmap v1122: 172 pgs: 172 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-07-12T15:43:35.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:35 smithi069 ceph-mon[94241]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2023-07-12T15:43:35.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:35 smithi069 ceph-mon[94241]: osdmap e728: 8 total, 8 up, 8 in 2023-07-12T15:43:35.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:35 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:36.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:43:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:43:36] "GET /metrics HTTP/1.1" 200 228575 "" "Prometheus/2.43.0" 2023-07-12T15:43:36.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:36 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:36.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-148","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:36.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:36 smithi145 ceph-mon[95801]: osdmap e729: 8 total, 8 up, 8 in 2023-07-12T15:43:36.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:36 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:43:36.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:36 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:36.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-148","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:36.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:36 smithi069 ceph-mon[102777]: osdmap e729: 8 total, 8 up, 8 in 2023-07-12T15:43:36.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:36 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:43:36.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:36 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:36.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-148","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:36.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:36 smithi069 ceph-mon[94241]: osdmap e729: 8 total, 8 up, 8 in 2023-07-12T15:43:36.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:36 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148", "force_nonempty": "--force-nonempty" }]: dispatch 2023-07-12T15:43:37.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:37 smithi145 ceph-mon[95801]: pgmap v1125: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-07-12T15:43:37.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:43:37.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:37 smithi145 ceph-mon[95801]: osdmap e730: 8 total, 8 up, 8 in 2023-07-12T15:43:37.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:37 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148"}]: dispatch 2023-07-12T15:43:37.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:37 smithi069 ceph-mon[102777]: pgmap v1125: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-07-12T15:43:37.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:43:37.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:37 smithi069 ceph-mon[102777]: osdmap e730: 8 total, 8 up, 8 in 2023-07-12T15:43:37.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:37 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148"}]: dispatch 2023-07-12T15:43:37.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:37 smithi069 ceph-mon[94241]: pgmap v1125: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-07-12T15:43:37.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148", "force_nonempty": "--force-nonempty" }]': finished 2023-07-12T15:43:37.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:37 smithi069 ceph-mon[94241]: osdmap e730: 8 total, 8 up, 8 in 2023-07-12T15:43:37.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:37 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148"}]: dispatch 2023-07-12T15:43:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148"}]': finished 2023-07-12T15:43:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:38 smithi145 ceph-mon[95801]: osdmap e731: 8 total, 8 up, 8 in 2023-07-12T15:43:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:38.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:38 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148"}]: dispatch 2023-07-12T15:43:38.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148"}]': finished 2023-07-12T15:43:38.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:38 smithi069 ceph-mon[94241]: osdmap e731: 8 total, 8 up, 8 in 2023-07-12T15:43:38.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:38.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:38 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148"}]: dispatch 2023-07-12T15:43:38.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148"}]': finished 2023-07-12T15:43:38.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:38 smithi069 ceph-mon[102777]: osdmap e731: 8 total, 8 up, 8 in 2023-07-12T15:43:38.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:38.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:38 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-148"}]: dispatch 2023-07-12T15:43:39.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:39 smithi145 ceph-mon[95801]: pgmap v1128: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:43:39.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:39 smithi145 ceph-mon[95801]: osdmap e732: 8 total, 8 up, 8 in 2023-07-12T15:43:39.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:39 smithi069 ceph-mon[102777]: pgmap v1128: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:43:39.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:39 smithi069 ceph-mon[102777]: osdmap e732: 8 total, 8 up, 8 in 2023-07-12T15:43:39.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:39 smithi069 ceph-mon[94241]: pgmap v1128: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:43:39.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:39 smithi069 ceph-mon[94241]: osdmap e732: 8 total, 8 up, 8 in 2023-07-12T15:43:40.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:40 smithi145 ceph-mon[95801]: osdmap e733: 8 total, 8 up, 8 in 2023-07-12T15:43:40.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:40 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:40.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:40 smithi069 ceph-mon[102777]: osdmap e733: 8 total, 8 up, 8 in 2023-07-12T15:43:40.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:40 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:40.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:40 smithi069 ceph-mon[94241]: osdmap e733: 8 total, 8 up, 8 in 2023-07-12T15:43:40.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:40 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:41 smithi145 ceph-mon[95801]: pgmap v1131: 204 pgs: 3 creating+peering, 29 unknown, 172 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:43:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-150","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:41 smithi145 ceph-mon[95801]: osdmap e734: 8 total, 8 up, 8 in 2023-07-12T15:43:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:41 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-150"}]: dispatch 2023-07-12T15:43:41.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:41 smithi069 ceph-mon[102777]: pgmap v1131: 204 pgs: 3 creating+peering, 29 unknown, 172 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:43:41.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-150","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:41.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:41 smithi069 ceph-mon[102777]: osdmap e734: 8 total, 8 up, 8 in 2023-07-12T15:43:41.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:41.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:41 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-150"}]: dispatch 2023-07-12T15:43:41.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:41 smithi069 ceph-mon[94241]: pgmap v1131: 204 pgs: 3 creating+peering, 29 unknown, 172 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:43:41.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-150","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:41.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:41 smithi069 ceph-mon[94241]: osdmap e734: 8 total, 8 up, 8 in 2023-07-12T15:43:41.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:41.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:41 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-150"}]: dispatch 2023-07-12T15:43:42.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:42 smithi145 ceph-mon[95801]: osdmap e735: 8 total, 8 up, 8 in 2023-07-12T15:43:42.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:42 smithi069 ceph-mon[102777]: osdmap e735: 8 total, 8 up, 8 in 2023-07-12T15:43:42.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:42 smithi069 ceph-mon[94241]: osdmap e735: 8 total, 8 up, 8 in 2023-07-12T15:43:43.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:43 smithi145 ceph-mon[95801]: pgmap v1134: 172 pgs: 172 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:43:43.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:43 smithi145 ceph-mon[95801]: osdmap e736: 8 total, 8 up, 8 in 2023-07-12T15:43:43.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:43.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:43 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:43:43.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:43 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:43:43.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:43 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-152","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:43.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:43 smithi145 ceph-mon[95801]: osdmap e737: 8 total, 8 up, 8 in 2023-07-12T15:43:43.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[94241]: pgmap v1134: 172 pgs: 172 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:43:43.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[94241]: osdmap e736: 8 total, 8 up, 8 in 2023-07-12T15:43:43.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:43.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:43:43.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:43:43.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-152","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:43.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[94241]: osdmap e737: 8 total, 8 up, 8 in 2023-07-12T15:43:43.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[102777]: pgmap v1134: 172 pgs: 172 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2023-07-12T15:43:43.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[102777]: osdmap e736: 8 total, 8 up, 8 in 2023-07-12T15:43:43.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:43.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:43:43.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:43:43.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-152","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:43.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:43 smithi069 ceph-mon[102777]: osdmap e737: 8 total, 8 up, 8 in 2023-07-12T15:43:45.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:44 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-152"}]: dispatch 2023-07-12T15:43:45.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:44 smithi145 ceph-mon[95801]: osdmap e738: 8 total, 8 up, 8 in 2023-07-12T15:43:45.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:44 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-152"}]: dispatch 2023-07-12T15:43:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:44 smithi069 ceph-mon[102777]: osdmap e738: 8 total, 8 up, 8 in 2023-07-12T15:43:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:44 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-152"}]: dispatch 2023-07-12T15:43:45.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:44 smithi069 ceph-mon[94241]: osdmap e738: 8 total, 8 up, 8 in 2023-07-12T15:43:45.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:43:45 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:43:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:45 smithi145 ceph-mon[95801]: pgmap v1137: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:45 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:45 smithi145 ceph-mon[95801]: osdmap e739: 8 total, 8 up, 8 in 2023-07-12T15:43:46.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:45 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:46.148 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:45 smithi069 ceph-mon[102777]: pgmap v1137: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:46.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:45 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:46.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:45 smithi069 ceph-mon[102777]: osdmap e739: 8 total, 8 up, 8 in 2023-07-12T15:43:46.149 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:45 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:46.149 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:45 smithi069 ceph-mon[94241]: pgmap v1137: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:46.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:45 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:46.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:45 smithi069 ceph-mon[94241]: osdmap e739: 8 total, 8 up, 8 in 2023-07-12T15:43:46.150 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:45 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-07-12T15:43:46.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:43:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:43:46] "GET /metrics HTTP/1.1" 200 228538 "" "Prometheus/2.43.0" 2023-07-12T15:43:47.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:47 smithi145 ceph-mon[95801]: pgmap v1140: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2023-07-12T15:43:47.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-154","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:47.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:47 smithi145 ceph-mon[95801]: osdmap e740: 8 total, 8 up, 8 in 2023-07-12T15:43:47.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-111","var": "dedup_tier","val": "test-rados-api-smithi069-133790-154"}]: dispatch 2023-07-12T15:43:47.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:47.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-154"}]: dispatch 2023-07-12T15:43:47.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:47 smithi069 ceph-mon[94241]: pgmap v1140: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2023-07-12T15:43:47.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-154","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:47.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:47 smithi069 ceph-mon[94241]: osdmap e740: 8 total, 8 up, 8 in 2023-07-12T15:43:47.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-111","var": "dedup_tier","val": "test-rados-api-smithi069-133790-154"}]: dispatch 2023-07-12T15:43:47.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:47.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-154"}]: dispatch 2023-07-12T15:43:47.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:47 smithi069 ceph-mon[102777]: pgmap v1140: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2023-07-12T15:43:47.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-133790-154","app": "rados","yes_i_really_mean_it": true}]': finished 2023-07-12T15:43:47.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:47 smithi069 ceph-mon[102777]: osdmap e740: 8 total, 8 up, 8 in 2023-07-12T15:43:47.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi069-133790-111","var": "dedup_tier","val": "test-rados-api-smithi069-133790-154"}]: dispatch 2023-07-12T15:43:47.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:47.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi069-133790-111", "tierpool": "test-rados-api-smithi069-133790-154"}]: dispatch 2023-07-12T15:43:48.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:48 smithi145 ceph-mon[95801]: osdmap e741: 8 total, 8 up, 8 in 2023-07-12T15:43:48.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:48 smithi069 ceph-mon[102777]: osdmap e741: 8 total, 8 up, 8 in 2023-07-12T15:43:48.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:48 smithi069 ceph-mon[94241]: osdmap e741: 8 total, 8 up, 8 in 2023-07-12T15:43:49.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:49 smithi145 ceph-mon[95801]: pgmap v1143: 172 pgs: 172 active+clean; 4.4 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2023-07-12T15:43:49.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:49 smithi145 ceph-mon[95801]: osdmap e742: 8 total, 8 up, 8 in 2023-07-12T15:43:49.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:49.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:49 smithi069 ceph-mon[102777]: pgmap v1143: 172 pgs: 172 active+clean; 4.4 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2023-07-12T15:43:49.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:49 smithi069 ceph-mon[102777]: osdmap e742: 8 total, 8 up, 8 in 2023-07-12T15:43:49.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:49.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:49 smithi069 ceph-mon[94241]: pgmap v1143: 172 pgs: 172 active+clean; 4.4 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2023-07-12T15:43:49.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:49 smithi069 ceph-mon[94241]: osdmap e742: 8 total, 8 up, 8 in 2023-07-12T15:43:49.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:50.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8052 ms) 2023-07-12T15:43:50.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2023-07-12T15:43:50.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (12767 ms) 2023-07-12T15:43:50.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2023-07-12T15:43:50.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8072 ms) 2023-07-12T15:43:50.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2023-07-12T15:43:50.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13117 ms) 2023-07-12T15:43:50.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2023-07-12T15:43:50.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8000 ms) 2023-07-12T15:43:50.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2023-07-12T15:43:50.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8040 ms) 2023-07-12T15:43:50.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2023-07-12T15:43:50.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: hmm, no HitSet yet 2023-07-12T15:43:50.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: ok, hit_set contains 255:602f83fe:::foo:head 2023-07-12T15:43:50.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9057 ms) 2023-07-12T15:43:50.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2023-07-12T15:43:50.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176547,0 2023-07-12T15:43:50.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first is 1689176547 2023-07-12T15:43:50.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176547,0 2023-07-12T15:43:50.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176547,0 2023-07-12T15:43:50.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176547,1689176549,1689176550,0 2023-07-12T15:43:50.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176547,1689176549,1689176550,0 2023-07-12T15:43:50.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176547,1689176549,1689176550,0 2023-07-12T15:43:50.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176547,1689176549,1689176550,1689176552,1689176553,0 2023-07-12T15:43:50.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176547,1689176549,1689176550,1689176552,1689176553,0 2023-07-12T15:43:50.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176547,1689176549,1689176550,1689176552,1689176553,0 2023-07-12T15:43:50.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176547,1689176549,1689176550,1689176552,1689176553,1689176555,1689176556,0 2023-07-12T15:43:50.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176547,1689176549,1689176550,1689176552,1689176553,1689176555,1689176556,0 2023-07-12T15:43:50.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176547,1689176549,1689176550,1689176552,1689176553,1689176555,1689176556,0 2023-07-12T15:43:50.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1689176550,1689176552,1689176553,1689176555,1689176556,1689176558,1689176559,0 2023-07-12T15:43:50.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first now 1689176550, trimmed 2023-07-12T15:43:50.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20279 ms) 2023-07-12T15:43:50.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2023-07-12T15:43:50.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: foo0 2023-07-12T15:43:50.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-07-12T15:43:50.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14070 ms) 2023-07-12T15:43:50.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2023-07-12T15:43:50.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17948 ms) 2023-07-12T15:43:50.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2023-07-12T15:43:50.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22183 ms) 2023-07-12T15:43:50.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2023-07-12T15:43:50.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5038 ms) 2023-07-12T15:43:50.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2023-07-12T15:43:50.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3019 ms) 2023-07-12T15:43:50.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2023-07-12T15:43:50.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3011 ms) 2023-07-12T15:43:50.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2023-07-12T15:43:50.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3008 ms) 2023-07-12T15:43:50.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (231205 ms total) 2023-07-12T15:43:50.600 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2023-07-12T15:43:50.601 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] Global test environment tear-down 2023-07-12T15:43:50.601 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (849858 ms total) 2023-07-12T15:43:50.601 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ PASSED ] 77 tests. 2023-07-12T15:43:50.601 INFO:tasks.workunit.client.0.smithi069.stderr:+ exit 0 2023-07-12T15:43:50.602 INFO:tasks.workunit.client.0.smithi069.stderr:+ cleanup 2023-07-12T15:43:50.602 INFO:tasks.workunit.client.0.smithi069.stderr:+ pkill -P 133609 2023-07-12T15:43:50.618 INFO:tasks.workunit.client.0.smithi069.stderr:+ true 2023-07-12T15:43:50.619 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-07-12T15:43:50.619 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-07-12T15:43:50.666 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2023-07-12T15:43:50.667 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2023-07-12T15:43:50.667 DEBUG:teuthology.orchestra.run.smithi069:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b1bd054042b7797ba5b70d145a1da31693dda494 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2023-07-12T15:43:50.689 INFO:tasks.workunit.client.0.smithi069.stderr:++ uuidgen 2023-07-12T15:43:50.690 INFO:tasks.workunit.client.0.smithi069.stderr:+ p=e6659867-2dff-445e-9a1e-87d1f41b1a67 2023-07-12T15:43:50.690 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool create e6659867-2dff-445e-9a1e-87d1f41b1a67 12 2023-07-12T15:43:50.792 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.790+0000 7f32812ef700 1 -- 172.21.15.69:0/3765306522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f327c0ff300 msgr2=0x7f327c0ff6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:50.792 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.790+0000 7f32812ef700 1 --2- 172.21.15.69:0/3765306522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f327c0ff300 0x7f327c0ff6d0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f326c009990 tx=0x7f326c009ca0 comp rx=0 tx=0).stop 2023-07-12T15:43:50.792 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.790+0000 7f32812ef700 1 -- 172.21.15.69:0/3765306522 shutdown_connections 2023-07-12T15:43:50.793 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.790+0000 7f32812ef700 1 --2- 172.21.15.69:0/3765306522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c10abc0 0x7f327c10cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:50.793 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.790+0000 7f32812ef700 1 --2- 172.21.15.69:0/3765306522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f327c0ff300 0x7f327c0ff6d0 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:50.793 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.790+0000 7f32812ef700 1 --2- 172.21.15.69:0/3765306522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f327c0ffc10 0x7f327c10a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:50.793 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.790+0000 7f32812ef700 1 -- 172.21.15.69:0/3765306522 >> 172.21.15.69:0/3765306522 conn(0x7f327c074890 msgr2=0x7f327c074c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:43:50.793 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.791+0000 7f32812ef700 1 -- 172.21.15.69:0/3765306522 shutdown_connections 2023-07-12T15:43:50.793 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.791+0000 7f32812ef700 1 -- 172.21.15.69:0/3765306522 wait complete. 2023-07-12T15:43:50.795 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.791+0000 7f32812ef700 1 Processor -- start 2023-07-12T15:43:50.795 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f32812ef700 1 -- start start 2023-07-12T15:43:50.795 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f32812ef700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f327c0ff300 0x7f327c19b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:43:50.796 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f32812ef700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c0ffc10 0x7f327c19bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:43:50.796 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f32812ef700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f327c10abc0 0x7f327c195730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:43:50.796 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f32812ef700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c195de0 con 0x7f327c0ffc10 2023-07-12T15:43:50.796 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f32812ef700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c195f50 con 0x7f327c0ff300 2023-07-12T15:43:50.796 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f32812ef700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327c196110 con 0x7f327c10abc0 2023-07-12T15:43:50.796 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327a7fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c0ffc10 0x7f327c19bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:43:50.796 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327b7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f327c10abc0 0x7f327c195730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:43:50.797 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327b7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f327c10abc0 0x7f327c195730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:40952/0 (socket says 172.21.15.69:40952) 2023-07-12T15:43:50.797 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327a7fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c0ffc10 0x7f327c19bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47522/0 (socket says 172.21.15.69:47522) 2023-07-12T15:43:50.797 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327b7fe700 1 -- 172.21.15.69:0/922629234 learned_addr learned my addr 172.21.15.69:0/922629234 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:43:50.797 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327affd700 1 --2- 172.21.15.69:0/922629234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f327c0ff300 0x7f327c19b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:43:50.797 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327b7fe700 1 -- 172.21.15.69:0/922629234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f327c0ff300 msgr2=0x7f327c19b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:50.797 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327b7fe700 1 --2- 172.21.15.69:0/922629234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f327c0ff300 0x7f327c19b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:50.797 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327b7fe700 1 -- 172.21.15.69:0/922629234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c0ffc10 msgr2=0x7f327c19bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:50.798 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327b7fe700 1 --2- 172.21.15.69:0/922629234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c0ffc10 0x7f327c19bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:50.798 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327b7fe700 1 -- 172.21.15.69:0/922629234 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3274009c40 con 0x7f327c10abc0 2023-07-12T15:43:50.798 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327a7fc700 1 --2- 172.21.15.69:0/922629234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c0ffc10 0x7f327c19bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:43:50.798 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327affd700 1 --2- 172.21.15.69:0/922629234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f327c0ff300 0x7f327c19b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:43:50.798 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f327b7fe700 1 --2- 172.21.15.69:0/922629234 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f327c10abc0 0x7f327c195730 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f327400c110 tx=0x7f327400c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:43:50.798 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f3263fff700 1 -- 172.21.15.69:0/922629234 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274017070 con 0x7f327c10abc0 2023-07-12T15:43:50.798 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.792+0000 7f32812ef700 1 -- 172.21.15.69:0/922629234 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326c0095f0 con 0x7f327c10abc0 2023-07-12T15:43:50.798 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.793+0000 7f3263fff700 1 -- 172.21.15.69:0/922629234 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f327400f040 con 0x7f327c10abc0 2023-07-12T15:43:50.799 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.793+0000 7f3263fff700 1 -- 172.21.15.69:0/922629234 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327400e050 con 0x7f327c10abc0 2023-07-12T15:43:50.799 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.793+0000 7f32812ef700 1 -- 172.21.15.69:0/922629234 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327c1a24f0 con 0x7f327c10abc0 2023-07-12T15:43:50.799 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.794+0000 7f32812ef700 1 -- 172.21.15.69:0/922629234 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f327c067e30 con 0x7f327c10abc0 2023-07-12T15:43:50.799 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.794+0000 7f3263fff700 1 -- 172.21.15.69:0/922629234 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f327400cad0 con 0x7f327c10abc0 2023-07-12T15:43:50.800 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.794+0000 7f3263fff700 1 --2- 172.21.15.69:0/922629234 >> v2:172.21.15.69:6800/1242716914 conn(0x7f326406c810 0x7f326406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:43:50.800 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.794+0000 7f3263fff700 1 -- 172.21.15.69:0/922629234 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(744..744 src has 1..744) v4 ==== 7283+0+0 (secure 0 0 0) 0x7f327408b8e0 con 0x7f327c10abc0 2023-07-12T15:43:50.800 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.794+0000 7f327affd700 1 --2- 172.21.15.69:0/922629234 >> v2:172.21.15.69:6800/1242716914 conn(0x7f326406c810 0x7f326406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:43:50.800 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.795+0000 7f327affd700 1 --2- 172.21.15.69:0/922629234 >> v2:172.21.15.69:6800/1242716914 conn(0x7f326406c810 0x7f326406ecc0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f326c00b2e0 tx=0x7f326c005fb0 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:43:50.800 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.799+0000 7f3263fff700 1 -- 172.21.15.69:0/922629234 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3274059330 con 0x7f327c10abc0 2023-07-12T15:43:50.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:43:50.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:50 smithi145 ceph-mon[95801]: osdmap e743: 8 total, 8 up, 8 in 2023-07-12T15:43:50.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:50 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:50.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:43:50.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:50 smithi069 ceph-mon[102777]: osdmap e743: 8 total, 8 up, 8 in 2023-07-12T15:43:50.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:50 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:50.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:43:50.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:50 smithi069 ceph-mon[94241]: osdmap e743: 8 total, 8 up, 8 in 2023-07-12T15:43:50.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:50 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-111"}]: dispatch 2023-07-12T15:43:50.991 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:50.990+0000 7f32812ef700 1 -- 172.21.15.69:0/922629234 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12} v 0) v1 -- 0x7f327c1a26e0 con 0x7f327c10abc0 2023-07-12T15:43:51.609 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.607+0000 7f3263fff700 1 -- 172.21.15.69:0/922629234 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]=0 pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' created v745) v1 ==== 176+0+0 (secure 0 0 0) 0x7f3274055070 con 0x7f327c10abc0 2023-07-12T15:43:51.733 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.732+0000 7f32812ef700 1 -- 172.21.15.69:0/922629234 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12} v 0) v1 -- 0x7f327c196c60 con 0x7f327c10abc0 2023-07-12T15:43:51.734 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.733+0000 7f3263fff700 1 -- 172.21.15.69:0/922629234 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]=0 pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' already exists v745) v1 ==== 183+0+0 (secure 0 0 0) 0x7f3274058ec0 con 0x7f327c10abc0 2023-07-12T15:43:51.736 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.735+0000 7f32812ef700 1 -- 172.21.15.69:0/922629234 >> v2:172.21.15.69:6800/1242716914 conn(0x7f326406c810 msgr2=0x7f326406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:51.737 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.735+0000 7f32812ef700 1 --2- 172.21.15.69:0/922629234 >> v2:172.21.15.69:6800/1242716914 conn(0x7f326406c810 0x7f326406ecc0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f326c00b2e0 tx=0x7f326c005fb0 comp rx=0 tx=0).stop 2023-07-12T15:43:51.737 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.735+0000 7f32812ef700 1 -- 172.21.15.69:0/922629234 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f327c10abc0 msgr2=0x7f327c195730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:51.737 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.735+0000 7f32812ef700 1 --2- 172.21.15.69:0/922629234 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f327c10abc0 0x7f327c195730 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f327400c110 tx=0x7f327400c420 comp rx=0 tx=0).stop 2023-07-12T15:43:51.737 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.735+0000 7f32812ef700 1 -- 172.21.15.69:0/922629234 shutdown_connections 2023-07-12T15:43:51.737 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.735+0000 7f32812ef700 1 --2- 172.21.15.69:0/922629234 >> v2:172.21.15.69:6800/1242716914 conn(0x7f326406c810 0x7f326406ecc0 unknown :-1 s=CLOSED pgs=3212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:51.738 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.735+0000 7f32812ef700 1 --2- 172.21.15.69:0/922629234 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f327c10abc0 0x7f327c195730 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:51.738 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.735+0000 7f32812ef700 1 --2- 172.21.15.69:0/922629234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f327c0ff300 0x7f327c19b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:51.738 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.735+0000 7f32812ef700 1 --2- 172.21.15.69:0/922629234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c0ffc10 0x7f327c19bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:51.738 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.735+0000 7f32812ef700 1 -- 172.21.15.69:0/922629234 >> 172.21.15.69:0/922629234 conn(0x7f327c074890 msgr2=0x7f327c0fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:43:51.738 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.735+0000 7f32812ef700 1 -- 172.21.15.69:0/922629234 shutdown_connections 2023-07-12T15:43:51.738 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.735+0000 7f32812ef700 1 -- 172.21.15.69:0/922629234 wait complete. 2023-07-12T15:43:51.739 INFO:tasks.workunit.client.0.smithi069.stderr:pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' already exists 2023-07-12T15:43:51.747 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool set-quota e6659867-2dff-445e-9a1e-87d1f41b1a67 max_objects 10 2023-07-12T15:43:51.849 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.847+0000 7f03ed233700 1 -- 172.21.15.69:0/1203459076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03e8109990 msgr2=0x7f03e810bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:51.849 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.847+0000 7f03ed233700 1 --2- 172.21.15.69:0/1203459076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03e8109990 0x7f03e810bd70 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f03e0009f90 tx=0x7f03e000b330 comp rx=0 tx=0).stop 2023-07-12T15:43:51.850 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.847+0000 7f03ed233700 1 -- 172.21.15.69:0/1203459076 shutdown_connections 2023-07-12T15:43:51.850 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.847+0000 7f03ed233700 1 --2- 172.21.15.69:0/1203459076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03e8109990 0x7f03e810bd70 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:51.850 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.847+0000 7f03ed233700 1 --2- 172.21.15.69:0/1203459076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03e81003e0 0x7f03e81007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:51.851 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.847+0000 7f03ed233700 1 --2- 172.21.15.69:0/1203459076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03e8100cf0 0x7f03e8109450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:51.851 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.847+0000 7f03ed233700 1 -- 172.21.15.69:0/1203459076 >> 172.21.15.69:0/1203459076 conn(0x7f03e80748c0 msgr2=0x7f03e8074cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:43:51.851 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.847+0000 7f03ed233700 1 -- 172.21.15.69:0/1203459076 shutdown_connections 2023-07-12T15:43:51.851 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.847+0000 7f03ed233700 1 -- 172.21.15.69:0/1203459076 wait complete. 2023-07-12T15:43:51.851 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03ed233700 1 Processor -- start 2023-07-12T15:43:51.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:51 smithi145 ceph-mon[95801]: pgmap v1146: 164 pgs: 164 active+clean; 582 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:51.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:43:51.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:51 smithi145 ceph-mon[95801]: osdmap e744: 8 total, 8 up, 8 in 2023-07-12T15:43:51.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:51 smithi145 ceph-mon[95801]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:43:51.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:51 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/922629234' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]: dispatch 2023-07-12T15:43:51.853 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:51 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]: dispatch 2023-07-12T15:43:51.855 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03ed233700 1 -- start start 2023-07-12T15:43:51.855 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03ed233700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03e81003e0 0x7f03e819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:43:51.855 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03ed233700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03e8100cf0 0x7f03e819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:43:51.855 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03ed233700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03e8109990 0x7f03e8194760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:43:51.855 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03ed233700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e8194dd0 con 0x7f03e8100cf0 2023-07-12T15:43:51.855 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03ed233700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e8194f40 con 0x7f03e8109990 2023-07-12T15:43:51.856 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03ed233700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e8195100 con 0x7f03e81003e0 2023-07-12T15:43:51.856 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03e6d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03e81003e0 0x7f03e819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:43:51.856 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03e659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03e8100cf0 0x7f03e819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:43:51.856 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03e6d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03e81003e0 0x7f03e819a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:40962/0 (socket says 172.21.15.69:40962) 2023-07-12T15:43:51.856 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03e659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03e8100cf0 0x7f03e819ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47534/0 (socket says 172.21.15.69:47534) 2023-07-12T15:43:51.856 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03e6d9d700 1 -- 172.21.15.69:0/3285115560 learned_addr learned my addr 172.21.15.69:0/3285115560 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:43:51.857 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.848+0000 7f03e759e700 1 --2- 172.21.15.69:0/3285115560 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03e8109990 0x7f03e8194760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:43:51.857 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.849+0000 7f03e6d9d700 1 -- 172.21.15.69:0/3285115560 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03e8109990 msgr2=0x7f03e8194760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:51.857 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.849+0000 7f03e6d9d700 1 --2- 172.21.15.69:0/3285115560 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03e8109990 0x7f03e8194760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:51.857 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.849+0000 7f03e6d9d700 1 -- 172.21.15.69:0/3285115560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03e8100cf0 msgr2=0x7f03e819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:51.857 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.849+0000 7f03e6d9d700 1 --2- 172.21.15.69:0/3285115560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03e8100cf0 0x7f03e819ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:51.857 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.849+0000 7f03e6d9d700 1 -- 172.21.15.69:0/3285115560 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc0095f0 con 0x7f03e81003e0 2023-07-12T15:43:51.857 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.849+0000 7f03e659c700 1 --2- 172.21.15.69:0/3285115560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03e8100cf0 0x7f03e819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:43:51.858 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.849+0000 7f03e6d9d700 1 --2- 172.21.15.69:0/3285115560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03e81003e0 0x7f03e819a6f0 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00ec10 tx=0x7f03dc00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:43:51.858 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.849+0000 7f03cffff700 1 -- 172.21.15.69:0/3285115560 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc00cc90 con 0x7f03e81003e0 2023-07-12T15:43:51.858 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.849+0000 7f03ed233700 1 -- 172.21.15.69:0/3285115560 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e0009c40 con 0x7f03e81003e0 2023-07-12T15:43:51.858 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.849+0000 7f03cffff700 1 -- 172.21.15.69:0/3285115560 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f03dc00cdf0 con 0x7f03e81003e0 2023-07-12T15:43:51.858 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.849+0000 7f03cffff700 1 -- 172.21.15.69:0/3285115560 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc003e90 con 0x7f03e81003e0 2023-07-12T15:43:51.858 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.849+0000 7f03ed233700 1 -- 172.21.15.69:0/3285115560 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e81955d0 con 0x7f03e81003e0 2023-07-12T15:43:51.858 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.850+0000 7f03ed233700 1 -- 172.21.15.69:0/3285115560 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e8067150 con 0x7f03e81003e0 2023-07-12T15:43:51.858 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.850+0000 7f03cffff700 1 -- 172.21.15.69:0/3285115560 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f03dc0104b0 con 0x7f03e81003e0 2023-07-12T15:43:51.859 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.851+0000 7f03cffff700 1 --2- 172.21.15.69:0/3285115560 >> v2:172.21.15.69:6800/1242716914 conn(0x7f03d006c810 0x7f03d006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:43:51.859 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.851+0000 7f03cffff700 1 -- 172.21.15.69:0/3285115560 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(745..745 src has 1..745) v4 ==== 7658+0+0 (secure 0 0 0) 0x7f03dc014070 con 0x7f03e81003e0 2023-07-12T15:43:51.859 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.851+0000 7f03e659c700 1 --2- 172.21.15.69:0/3285115560 >> v2:172.21.15.69:6800/1242716914 conn(0x7f03d006c810 0x7f03d006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:43:51.859 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.851+0000 7f03e659c700 1 --2- 172.21.15.69:0/3285115560 >> v2:172.21.15.69:6800/1242716914 conn(0x7f03d006c810 0x7f03d006ecc0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7f03e8195bd0 tx=0x7f03d8009190 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:43:51.859 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:51.855+0000 7f03cffff700 1 -- 172.21.15.69:0/3285115560 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f03dc059ed0 con 0x7f03e81003e0 2023-07-12T15:43:51.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:51 smithi069 ceph-mon[102777]: pgmap v1146: 164 pgs: 164 active+clean; 582 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:51.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:43:51.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:51 smithi069 ceph-mon[102777]: osdmap e744: 8 total, 8 up, 8 in 2023-07-12T15:43:51.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:51 smithi069 ceph-mon[102777]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:43:51.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:51 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/922629234' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]: dispatch 2023-07-12T15:43:51.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:51 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]: dispatch 2023-07-12T15:43:51.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:51 smithi069 ceph-mon[94241]: pgmap v1146: 164 pgs: 164 active+clean; 582 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:51.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1872647546' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi069-133790-111"}]': finished 2023-07-12T15:43:51.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:51 smithi069 ceph-mon[94241]: osdmap e744: 8 total, 8 up, 8 in 2023-07-12T15:43:51.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:51 smithi069 ceph-mon[94241]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-07-12T15:43:51.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:51 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/922629234' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]: dispatch 2023-07-12T15:43:51.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:51 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]: dispatch 2023-07-12T15:43:52.049 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:52.048+0000 7f03ed233700 1 -- 172.21.15.69:0/3285115560 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f03e81955d0 con 0x7f03e81003e0 2023-07-12T15:43:52.611 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:52.610+0000 7f03cffff700 1 -- 172.21.15.69:0/3285115560 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v746) v1 ==== 223+0+0 (secure 0 0 0) 0x7f03e81955d0 con 0x7f03e81003e0 2023-07-12T15:43:52.740 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:52.739+0000 7f03ed233700 1 -- 172.21.15.69:0/3285115560 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f03e81955d0 con 0x7f03e81003e0 2023-07-12T15:43:52.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]': finished 2023-07-12T15:43:52.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:52 smithi069 ceph-mon[102777]: osdmap e745: 8 total, 8 up, 8 in 2023-07-12T15:43:52.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/922629234' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]: dispatch 2023-07-12T15:43:52.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]: dispatch 2023-07-12T15:43:52.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:52 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3285115560' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:43:52.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:52 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:43:52.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]': finished 2023-07-12T15:43:52.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:52 smithi069 ceph-mon[94241]: osdmap e745: 8 total, 8 up, 8 in 2023-07-12T15:43:52.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/922629234' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]: dispatch 2023-07-12T15:43:52.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]: dispatch 2023-07-12T15:43:52.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:52 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3285115560' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:43:52.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:52 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:43:53.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]': finished 2023-07-12T15:43:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:52 smithi145 ceph-mon[95801]: osdmap e745: 8 total, 8 up, 8 in 2023-07-12T15:43:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/922629234' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]: dispatch 2023-07-12T15:43:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pg_num": 12}]: dispatch 2023-07-12T15:43:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:52 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3285115560' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:43:53.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:52 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:43:53.632 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.631+0000 7f03cffff700 1 -- 172.21.15.69:0/3285115560 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v747) v1 ==== 223+0+0 (secure 0 0 0) 0x7f03e81955d0 con 0x7f03e81003e0 2023-07-12T15:43:53.635 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.633+0000 7f03ed233700 1 -- 172.21.15.69:0/3285115560 >> v2:172.21.15.69:6800/1242716914 conn(0x7f03d006c810 msgr2=0x7f03d006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:53.635 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.633+0000 7f03ed233700 1 --2- 172.21.15.69:0/3285115560 >> v2:172.21.15.69:6800/1242716914 conn(0x7f03d006c810 0x7f03d006ecc0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7f03e8195bd0 tx=0x7f03d8009190 comp rx=0 tx=0).stop 2023-07-12T15:43:53.635 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.633+0000 7f03ed233700 1 -- 172.21.15.69:0/3285115560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03e81003e0 msgr2=0x7f03e819a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:53.636 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.633+0000 7f03ed233700 1 --2- 172.21.15.69:0/3285115560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03e81003e0 0x7f03e819a6f0 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00ec10 tx=0x7f03dc00ef20 comp rx=0 tx=0).stop 2023-07-12T15:43:53.636 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.634+0000 7f03ed233700 1 -- 172.21.15.69:0/3285115560 shutdown_connections 2023-07-12T15:43:53.636 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.634+0000 7f03ed233700 1 --2- 172.21.15.69:0/3285115560 >> v2:172.21.15.69:6800/1242716914 conn(0x7f03d006c810 0x7f03d006ecc0 unknown :-1 s=CLOSED pgs=3213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:53.636 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.634+0000 7f03ed233700 1 --2- 172.21.15.69:0/3285115560 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03e8109990 0x7f03e8194760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:53.636 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.634+0000 7f03ed233700 1 --2- 172.21.15.69:0/3285115560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03e81003e0 0x7f03e819a6f0 unknown :-1 s=CLOSED pgs=2731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:53.636 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.634+0000 7f03ed233700 1 --2- 172.21.15.69:0/3285115560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03e8100cf0 0x7f03e819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:53.636 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.634+0000 7f03ed233700 1 -- 172.21.15.69:0/3285115560 >> 172.21.15.69:0/3285115560 conn(0x7f03e80748c0 msgr2=0x7f03e80fc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:43:53.637 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.634+0000 7f03ed233700 1 -- 172.21.15.69:0/3285115560 shutdown_connections 2023-07-12T15:43:53.637 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.634+0000 7f03ed233700 1 -- 172.21.15.69:0/3285115560 wait complete. 2023-07-12T15:43:53.637 INFO:tasks.workunit.client.0.smithi069.stderr:set-quota max_objects = 10 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 2023-07-12T15:43:53.646 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool application enable e6659867-2dff-445e-9a1e-87d1f41b1a67 rados 2023-07-12T15:43:53.751 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.749+0000 7fb95390c700 1 -- 172.21.15.69:0/3550282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94c1053d0 msgr2=0x7fb94c10cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:53.751 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.749+0000 7fb95390c700 1 --2- 172.21.15.69:0/3550282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94c1053d0 0x7fb94c10cfa0 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7fb944009fc0 tx=0x7fb94400b330 comp rx=0 tx=0).stop 2023-07-12T15:43:53.751 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.749+0000 7fb95390c700 1 -- 172.21.15.69:0/3550282 shutdown_connections 2023-07-12T15:43:53.751 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.749+0000 7fb95390c700 1 --2- 172.21.15.69:0/3550282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94c1053d0 0x7fb94c10cfa0 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:53.751 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.749+0000 7fb95390c700 1 --2- 172.21.15.69:0/3550282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb94c100390 0x7fb94c100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:53.752 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.749+0000 7fb95390c700 1 --2- 172.21.15.69:0/3550282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb94c100d30 0x7fb94c104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:53.752 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.749+0000 7fb95390c700 1 -- 172.21.15.69:0/3550282 >> 172.21.15.69:0/3550282 conn(0x7fb94c0fbc40 msgr2=0x7fb94c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:43:53.752 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.749+0000 7fb95390c700 1 -- 172.21.15.69:0/3550282 shutdown_connections 2023-07-12T15:43:53.752 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.750+0000 7fb95390c700 1 -- 172.21.15.69:0/3550282 wait complete. 2023-07-12T15:43:53.754 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.750+0000 7fb95390c700 1 Processor -- start 2023-07-12T15:43:53.754 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.750+0000 7fb95390c700 1 -- start start 2023-07-12T15:43:53.754 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.750+0000 7fb95390c700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94c100390 0x7fb94c19b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:43:53.755 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.750+0000 7fb95390c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb94c100d30 0x7fb94c19bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:43:53.755 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.750+0000 7fb95390c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb94c1053d0 0x7fb94c195a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:43:53.755 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.750+0000 7fb95390c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c1960d0 con 0x7fb94c1053d0 2023-07-12T15:43:53.755 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.750+0000 7fb95390c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c196240 con 0x7fb94c100d30 2023-07-12T15:43:53.755 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.750+0000 7fb95390c700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c196400 con 0x7fb94c100390 2023-07-12T15:43:53.755 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.750+0000 7fb9516a8700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94c100390 0x7fb94c19b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:43:53.756 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.750+0000 7fb9516a8700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94c100390 0x7fb94c19b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:40976/0 (socket says 172.21.15.69:40976) 2023-07-12T15:43:53.756 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.750+0000 7fb9516a8700 1 -- 172.21.15.69:0/2521243143 learned_addr learned my addr 172.21.15.69:0/2521243143 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:43:53.756 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.750+0000 7fb951ea9700 1 --2- 172.21.15.69:0/2521243143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb94c1053d0 0x7fb94c195a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:43:53.756 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.751+0000 7fb950ea7700 1 --2- 172.21.15.69:0/2521243143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb94c100d30 0x7fb94c19bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:43:53.756 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.751+0000 7fb9516a8700 1 -- 172.21.15.69:0/2521243143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb94c100d30 msgr2=0x7fb94c19bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:53.756 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.751+0000 7fb9516a8700 1 --2- 172.21.15.69:0/2521243143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb94c100d30 0x7fb94c19bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:53.757 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.751+0000 7fb9516a8700 1 -- 172.21.15.69:0/2521243143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb94c1053d0 msgr2=0x7fb94c195a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:53.757 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.751+0000 7fb9516a8700 1 --2- 172.21.15.69:0/2521243143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb94c1053d0 0x7fb94c195a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:53.757 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.751+0000 7fb9516a8700 1 -- 172.21.15.69:0/2521243143 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9480095f0 con 0x7fb94c100390 2023-07-12T15:43:53.757 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.751+0000 7fb951ea9700 1 --2- 172.21.15.69:0/2521243143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb94c1053d0 0x7fb94c195a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:43:53.757 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.751+0000 7fb9516a8700 1 --2- 172.21.15.69:0/2521243143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94c100390 0x7fb94c19b9f0 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fb94800ec10 tx=0x7fb94800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:43:53.757 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.751+0000 7fb9427fc700 1 -- 172.21.15.69:0/2521243143 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94800cd40 con 0x7fb94c100390 2023-07-12T15:43:53.757 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.751+0000 7fb9427fc700 1 -- 172.21.15.69:0/2521243143 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fb9480044b0 con 0x7fb94c100390 2023-07-12T15:43:53.757 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.751+0000 7fb95390c700 1 -- 172.21.15.69:0/2521243143 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb944009c40 con 0x7fb94c100390 2023-07-12T15:43:53.758 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.751+0000 7fb9427fc700 1 -- 172.21.15.69:0/2521243143 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb948010430 con 0x7fb94c100390 2023-07-12T15:43:53.758 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.751+0000 7fb95390c700 1 -- 172.21.15.69:0/2521243143 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c1a2800 con 0x7fb94c100390 2023-07-12T15:43:53.758 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.752+0000 7fb95390c700 1 -- 172.21.15.69:0/2521243143 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94c1086f0 con 0x7fb94c100390 2023-07-12T15:43:53.758 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.752+0000 7fb9427fc700 1 -- 172.21.15.69:0/2521243143 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7fb948003fa0 con 0x7fb94c100390 2023-07-12T15:43:53.758 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.753+0000 7fb9427fc700 1 --2- 172.21.15.69:0/2521243143 >> v2:172.21.15.69:6800/1242716914 conn(0x7fb93806c810 0x7fb93806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:43:53.758 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.753+0000 7fb950ea7700 1 --2- 172.21.15.69:0/2521243143 >> v2:172.21.15.69:6800/1242716914 conn(0x7fb93806c810 0x7fb93806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:43:53.758 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.753+0000 7fb9427fc700 1 -- 172.21.15.69:0/2521243143 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(747..747 src has 1..747) v4 ==== 7658+0+0 (secure 0 0 0) 0x7fb948014070 con 0x7fb94c100390 2023-07-12T15:43:53.759 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.753+0000 7fb950ea7700 1 --2- 172.21.15.69:0/2521243143 >> v2:172.21.15.69:6800/1242716914 conn(0x7fb93806c810 0x7fb93806ecc0 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7fb93c005fd0 tx=0x7fb93c005ee0 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:43:53.759 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.757+0000 7fb9427fc700 1 -- 172.21.15.69:0/2521243143 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb94805a000 con 0x7fb94c100390 2023-07-12T15:43:53.948 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:53.947+0000 7fb95390c700 1 -- 172.21.15.69:0/2521243143 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"} v 0) v1 -- 0x7fb94c0679e0 con 0x7fb94c100390 2023-07-12T15:43:53.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:53 smithi069 ceph-mon[94241]: pgmap v1149: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:53.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]': finished 2023-07-12T15:43:53.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:53 smithi069 ceph-mon[94241]: osdmap e746: 8 total, 8 up, 8 in 2023-07-12T15:43:53.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:53 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3285115560' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:43:53.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:53 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:43:53.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:53 smithi069 ceph-mon[102777]: pgmap v1149: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:53.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]': finished 2023-07-12T15:43:53.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:53 smithi069 ceph-mon[102777]: osdmap e746: 8 total, 8 up, 8 in 2023-07-12T15:43:53.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:53 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3285115560' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:43:53.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:53 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:43:54.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:53 smithi145 ceph-mon[95801]: pgmap v1149: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:54.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]': finished 2023-07-12T15:43:54.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:53 smithi145 ceph-mon[95801]: osdmap e746: 8 total, 8 up, 8 in 2023-07-12T15:43:54.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:53 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3285115560' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:43:54.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:53 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:43:54.637 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:54.635+0000 7fb9427fc700 1 -- 172.21.15.69:0/2521243143 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]=0 enabled application 'rados' on pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' v748) v1 ==== 213+0+0 (secure 0 0 0) 0x7fb948059b90 con 0x7fb94c100390 2023-07-12T15:43:54.762 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:54.761+0000 7fb95390c700 1 -- 172.21.15.69:0/2521243143 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"} v 0) v1 -- 0x7fb94c1080d0 con 0x7fb94c100390 2023-07-12T15:43:54.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]': finished 2023-07-12T15:43:54.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:54 smithi069 ceph-mon[102777]: osdmap e747: 8 total, 8 up, 8 in 2023-07-12T15:43:54.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:54 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2521243143' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]: dispatch 2023-07-12T15:43:54.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:54 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]: dispatch 2023-07-12T15:43:54.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]': finished 2023-07-12T15:43:54.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:54 smithi069 ceph-mon[94241]: osdmap e747: 8 total, 8 up, 8 in 2023-07-12T15:43:54.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:54 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2521243143' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]: dispatch 2023-07-12T15:43:54.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:54 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]: dispatch 2023-07-12T15:43:55.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "10"}]': finished 2023-07-12T15:43:55.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:54 smithi145 ceph-mon[95801]: osdmap e747: 8 total, 8 up, 8 in 2023-07-12T15:43:55.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:54 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2521243143' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]: dispatch 2023-07-12T15:43:55.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:54 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]: dispatch 2023-07-12T15:43:55.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:43:55 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:43:55.643 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:55.642+0000 7fb9427fc700 1 -- 172.21.15.69:0/2521243143 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]=0 enabled application 'rados' on pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' v749) v1 ==== 213+0+0 (secure 0 0 0) 0x7fb948059d70 con 0x7fb94c100390 2023-07-12T15:43:55.647 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:55.645+0000 7fb95390c700 1 -- 172.21.15.69:0/2521243143 >> v2:172.21.15.69:6800/1242716914 conn(0x7fb93806c810 msgr2=0x7fb93806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:55.647 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:55.645+0000 7fb95390c700 1 --2- 172.21.15.69:0/2521243143 >> v2:172.21.15.69:6800/1242716914 conn(0x7fb93806c810 0x7fb93806ecc0 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7fb93c005fd0 tx=0x7fb93c005ee0 comp rx=0 tx=0).stop 2023-07-12T15:43:55.647 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:55.645+0000 7fb95390c700 1 -- 172.21.15.69:0/2521243143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94c100390 msgr2=0x7fb94c19b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:43:55.647 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:55.645+0000 7fb95390c700 1 --2- 172.21.15.69:0/2521243143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94c100390 0x7fb94c19b9f0 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fb94800ec10 tx=0x7fb94800ef20 comp rx=0 tx=0).stop 2023-07-12T15:43:55.647 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:55.645+0000 7fb95390c700 1 -- 172.21.15.69:0/2521243143 shutdown_connections 2023-07-12T15:43:55.647 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:55.645+0000 7fb95390c700 1 --2- 172.21.15.69:0/2521243143 >> v2:172.21.15.69:6800/1242716914 conn(0x7fb93806c810 0x7fb93806ecc0 unknown :-1 s=CLOSED pgs=3214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:55.648 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:55.645+0000 7fb95390c700 1 --2- 172.21.15.69:0/2521243143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb94c1053d0 0x7fb94c195a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:55.648 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:55.645+0000 7fb95390c700 1 --2- 172.21.15.69:0/2521243143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94c100390 0x7fb94c19b9f0 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:55.648 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:55.645+0000 7fb95390c700 1 --2- 172.21.15.69:0/2521243143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb94c100d30 0x7fb94c19bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:43:55.648 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:55.645+0000 7fb95390c700 1 -- 172.21.15.69:0/2521243143 >> 172.21.15.69:0/2521243143 conn(0x7fb94c0fbc40 msgr2=0x7fb94c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:43:55.648 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:55.645+0000 7fb95390c700 1 -- 172.21.15.69:0/2521243143 shutdown_connections 2023-07-12T15:43:55.648 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:43:55.645+0000 7fb95390c700 1 -- 172.21.15.69:0/2521243143 wait complete. 2023-07-12T15:43:55.649 INFO:tasks.workunit.client.0.smithi069.stderr:enabled application 'rados' on pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' 2023-07-12T15:43:55.660 INFO:tasks.workunit.client.0.smithi069.stderr:++ seq 1 10 2023-07-12T15:43:55.684 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:43:55.684 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put obj1 /etc/passwd 2023-07-12T15:43:55.723 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:43:55.723 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put obj2 /etc/passwd 2023-07-12T15:43:55.762 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:43:55.762 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put obj3 /etc/passwd 2023-07-12T15:43:55.798 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:43:55.798 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put obj4 /etc/passwd 2023-07-12T15:43:55.833 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:43:55.834 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put obj5 /etc/passwd 2023-07-12T15:43:55.867 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:43:55.868 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put obj6 /etc/passwd 2023-07-12T15:43:55.902 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:43:55.902 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put obj7 /etc/passwd 2023-07-12T15:43:55.936 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:43:55.936 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put obj8 /etc/passwd 2023-07-12T15:43:55.970 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:43:55.971 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put obj9 /etc/passwd 2023-07-12T15:43:55.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:55 smithi069 ceph-mon[102777]: pgmap v1152: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:55.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:55 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]': finished 2023-07-12T15:43:55.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:55 smithi069 ceph-mon[102777]: osdmap e748: 8 total, 8 up, 8 in 2023-07-12T15:43:55.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:55 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2521243143' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]: dispatch 2023-07-12T15:43:55.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:55 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]: dispatch 2023-07-12T15:43:55.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:55 smithi069 ceph-mon[94241]: pgmap v1152: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:55.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:55 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]': finished 2023-07-12T15:43:55.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:55 smithi069 ceph-mon[94241]: osdmap e748: 8 total, 8 up, 8 in 2023-07-12T15:43:55.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:55 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2521243143' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]: dispatch 2023-07-12T15:43:55.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:55 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]: dispatch 2023-07-12T15:43:56.005 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:43:56.005 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put obj10 /etc/passwd 2023-07-12T15:43:56.040 INFO:tasks.workunit.client.0.smithi069.stderr:+ sleep 30 2023-07-12T15:43:56.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:55 smithi145 ceph-mon[95801]: pgmap v1152: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2023-07-12T15:43:56.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:55 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]': finished 2023-07-12T15:43:56.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:55 smithi145 ceph-mon[95801]: osdmap e748: 8 total, 8 up, 8 in 2023-07-12T15:43:56.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:55 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2521243143' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]: dispatch 2023-07-12T15:43:56.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:55 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]: dispatch 2023-07-12T15:43:56.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:43:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:43:56] "GET /metrics HTTP/1.1" 200 228538 "" "Prometheus/2.43.0" 2023-07-12T15:43:56.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:56 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:56.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:56 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]': finished 2023-07-12T15:43:56.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:56 smithi069 ceph-mon[94241]: osdmap e749: 8 total, 8 up, 8 in 2023-07-12T15:43:56.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:56 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:56.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:56 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]': finished 2023-07-12T15:43:56.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:56 smithi069 ceph-mon[102777]: osdmap e749: 8 total, 8 up, 8 in 2023-07-12T15:43:57.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:56 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:43:57.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:56 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "app": "rados"}]': finished 2023-07-12T15:43:57.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:56 smithi145 ceph-mon[95801]: osdmap e749: 8 total, 8 up, 8 in 2023-07-12T15:43:57.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:57 smithi069 ceph-mon[102777]: pgmap v1155: 176 pgs: 176 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:43:57.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:57 smithi069 ceph-mon[94241]: pgmap v1155: 176 pgs: 176 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:43:58.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:57 smithi145 ceph-mon[95801]: pgmap v1155: 176 pgs: 176 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:43:58.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:58 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:43:58.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:58 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:43:59.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:58 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:43:59.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:43:59 smithi069 ceph-mon[102777]: pgmap v1156: 176 pgs: 176 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2023-07-12T15:43:59.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:43:59 smithi069 ceph-mon[94241]: pgmap v1156: 176 pgs: 176 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2023-07-12T15:44:00.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:43:59 smithi145 ceph-mon[95801]: pgmap v1156: 176 pgs: 176 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2023-07-12T15:44:01.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:01 smithi069 ceph-mon[102777]: pgmap v1157: 176 pgs: 176 active+clean; 689 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 17 KiB/s wr, 3 op/s 2023-07-12T15:44:01.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:01 smithi069 ceph-mon[94241]: pgmap v1157: 176 pgs: 176 active+clean; 689 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 17 KiB/s wr, 3 op/s 2023-07-12T15:44:02.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:01 smithi145 ceph-mon[95801]: pgmap v1157: 176 pgs: 176 active+clean; 689 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 17 KiB/s wr, 3 op/s 2023-07-12T15:44:02.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:02 smithi069 ceph-mon[102777]: pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' is full (reached quota's max_objects: 10) 2023-07-12T15:44:02.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:02 smithi069 ceph-mon[102777]: Health check failed: 1 pool(s) full (POOL_FULL) 2023-07-12T15:44:02.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:02 smithi069 ceph-mon[102777]: osdmap e750: 8 total, 8 up, 8 in 2023-07-12T15:44:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:02 smithi069 ceph-mon[94241]: pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' is full (reached quota's max_objects: 10) 2023-07-12T15:44:02.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:02 smithi069 ceph-mon[94241]: Health check failed: 1 pool(s) full (POOL_FULL) 2023-07-12T15:44:02.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:02 smithi069 ceph-mon[94241]: osdmap e750: 8 total, 8 up, 8 in 2023-07-12T15:44:03.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:02 smithi145 ceph-mon[95801]: pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' is full (reached quota's max_objects: 10) 2023-07-12T15:44:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:02 smithi145 ceph-mon[95801]: Health check failed: 1 pool(s) full (POOL_FULL) 2023-07-12T15:44:03.101 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:02 smithi145 ceph-mon[95801]: osdmap e750: 8 total, 8 up, 8 in 2023-07-12T15:44:03.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:03 smithi069 ceph-mon[102777]: pgmap v1159: 176 pgs: 176 active+clean; 689 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 673 B/s rd, 14 KiB/s wr, 1 op/s 2023-07-12T15:44:03.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:03 smithi069 ceph-mon[94241]: pgmap v1159: 176 pgs: 176 active+clean; 689 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 673 B/s rd, 14 KiB/s wr, 1 op/s 2023-07-12T15:44:04.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:03 smithi145 ceph-mon[95801]: pgmap v1159: 176 pgs: 176 active+clean; 689 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 673 B/s rd, 14 KiB/s wr, 1 op/s 2023-07-12T15:44:05.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:44:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:44:05.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:05 smithi069 ceph-mon[102777]: pgmap v1160: 176 pgs: 176 active+clean; 689 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 13 KiB/s wr, 1 op/s 2023-07-12T15:44:05.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:05 smithi069 ceph-mon[94241]: pgmap v1160: 176 pgs: 176 active+clean; 689 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 13 KiB/s wr, 1 op/s 2023-07-12T15:44:06.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:05 smithi145 ceph-mon[95801]: pgmap v1160: 176 pgs: 176 active+clean; 689 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 13 KiB/s wr, 1 op/s 2023-07-12T15:44:06.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:44:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:44:06] "GET /metrics HTTP/1.1" 200 226064 "" "Prometheus/2.43.0" 2023-07-12T15:44:06.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:06 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:06.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:06 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:07.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:06 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:07.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:07 smithi069 ceph-mon[102777]: pgmap v1161: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2023-07-12T15:44:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:07 smithi069 ceph-mon[94241]: pgmap v1161: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2023-07-12T15:44:08.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:07 smithi145 ceph-mon[95801]: pgmap v1161: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2023-07-12T15:44:09.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:09 smithi069 ceph-mon[102777]: pgmap v1162: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2023-07-12T15:44:09.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:09 smithi069 ceph-mon[94241]: pgmap v1162: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2023-07-12T15:44:10.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:09 smithi145 ceph-mon[95801]: pgmap v1162: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2023-07-12T15:44:11.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:11 smithi069 ceph-mon[102777]: pgmap v1163: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:44:11.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:11 smithi069 ceph-mon[94241]: pgmap v1163: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:44:12.100 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:11 smithi145 ceph-mon[95801]: pgmap v1163: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:44:13.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:12 smithi069 ceph-mon[102777]: pgmap v1164: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 995 B/s rd, 0 op/s 2023-07-12T15:44:13.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:12 smithi069 ceph-mon[94241]: pgmap v1164: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 995 B/s rd, 0 op/s 2023-07-12T15:44:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:12 smithi145 ceph-mon[95801]: pgmap v1164: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 995 B/s rd, 0 op/s 2023-07-12T15:44:14.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:14.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:14 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:44:14.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:14.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:14 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:44:14.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:14.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:14 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:44:15.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:44:15 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:44:15.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:15 smithi145 ceph-mon[95801]: pgmap v1165: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:15.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:15 smithi069 ceph-mon[102777]: pgmap v1165: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:15.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:15 smithi069 ceph-mon[94241]: pgmap v1165: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:16.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:44:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:44:16] "GET /metrics HTTP/1.1" 200 226085 "" "Prometheus/2.43.0" 2023-07-12T15:44:16.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:16 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:16.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:16 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:16.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:16 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:17.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:17 smithi145 ceph-mon[95801]: pgmap v1166: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:17.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:17 smithi069 ceph-mon[94241]: pgmap v1166: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:17.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:17 smithi069 ceph-mon[102777]: pgmap v1166: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:19.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:19 smithi145 ceph-mon[95801]: pgmap v1167: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:19.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:19 smithi069 ceph-mon[102777]: pgmap v1167: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:19.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:19 smithi069 ceph-mon[94241]: pgmap v1167: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:21.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:21 smithi145 ceph-mon[95801]: pgmap v1168: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:21.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:21 smithi069 ceph-mon[102777]: pgmap v1168: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:21 smithi069 ceph-mon[94241]: pgmap v1168: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:23.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:22 smithi069 ceph-mon[102777]: pgmap v1169: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:23.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:22 smithi069 ceph-mon[94241]: pgmap v1169: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:23.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:22 smithi145 ceph-mon[95801]: pgmap v1169: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:25.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:44:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:44:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:25 smithi145 ceph-mon[95801]: pgmap v1170: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:25.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:25 smithi069 ceph-mon[102777]: pgmap v1170: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:25.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:25 smithi069 ceph-mon[94241]: pgmap v1170: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:26.041 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=204153 2023-07-12T15:44:26.042 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool set-quota e6659867-2dff-445e-9a1e-87d1f41b1a67 max_objects 100 2023-07-12T15:44:26.042 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put onemore /etc/passwd 2023-07-12T15:44:26.159 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.157+0000 7f7febbd2700 1 -- 172.21.15.69:0/3646497594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe40ff2b0 msgr2=0x7f7fe40ff680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:44:26.159 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.157+0000 7f7febbd2700 1 --2- 172.21.15.69:0/3646497594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe40ff2b0 0x7f7fe40ff680 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f7fe00099c0 tx=0x7f7fe0009cd0 comp rx=0 tx=0).stop 2023-07-12T15:44:26.160 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.157+0000 7f7febbd2700 1 -- 172.21.15.69:0/3646497594 shutdown_connections 2023-07-12T15:44:26.160 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.157+0000 7f7febbd2700 1 --2- 172.21.15.69:0/3646497594 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7fe41042f0 0x7f7fe410bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:26.160 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.157+0000 7f7febbd2700 1 --2- 172.21.15.69:0/3646497594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe40ff2b0 0x7f7fe40ff680 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:26.160 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.157+0000 7f7febbd2700 1 --2- 172.21.15.69:0/3646497594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fe40ffc50 0x7f7fe4103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:26.160 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.157+0000 7f7febbd2700 1 -- 172.21.15.69:0/3646497594 >> 172.21.15.69:0/3646497594 conn(0x7f7fe4075e10 msgr2=0x7f7fe4076210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:44:26.160 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.157+0000 7f7febbd2700 1 -- 172.21.15.69:0/3646497594 shutdown_connections 2023-07-12T15:44:26.160 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.157+0000 7f7febbd2700 1 -- 172.21.15.69:0/3646497594 wait complete. 2023-07-12T15:44:26.163 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.158+0000 7f7febbd2700 1 Processor -- start 2023-07-12T15:44:26.163 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.158+0000 7f7febbd2700 1 -- start start 2023-07-12T15:44:26.163 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7febbd2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7fe40ff2b0 0x7f7fe419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:44:26.163 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7febbd2700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fe40ffc50 0x7f7fe419abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:44:26.163 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7febbd2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe41042f0 0x7f7fe41946f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:44:26.163 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7febbd2700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe4194d60 con 0x7f7fe41042f0 2023-07-12T15:44:26.164 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7febbd2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe4194ed0 con 0x7f7fe40ff2b0 2023-07-12T15:44:26.164 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7febbd2700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe4195090 con 0x7f7fe40ffc50 2023-07-12T15:44:26.164 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fe916d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fe40ffc50 0x7f7fe419abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:44:26.164 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fe916d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fe40ffc50 0x7f7fe419abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41236/0 (socket says 172.21.15.69:41236) 2023-07-12T15:44:26.164 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fea16f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe41042f0 0x7f7fe41946f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:44:26.164 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fe916d700 1 -- 172.21.15.69:0/1759452589 learned_addr learned my addr 172.21.15.69:0/1759452589 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:44:26.164 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fe996e700 1 --2- 172.21.15.69:0/1759452589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7fe40ff2b0 0x7f7fe419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:44:26.165 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fe916d700 1 -- 172.21.15.69:0/1759452589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7fe40ff2b0 msgr2=0x7f7fe419a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:44:26.165 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fe916d700 1 --2- 172.21.15.69:0/1759452589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7fe40ff2b0 0x7f7fe419a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:26.165 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fe916d700 1 -- 172.21.15.69:0/1759452589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe41042f0 msgr2=0x7f7fe41946f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:44:26.165 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fe916d700 1 --2- 172.21.15.69:0/1759452589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe41042f0 0x7f7fe41946f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:26.165 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fe916d700 1 -- 172.21.15.69:0/1759452589 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd4009520 con 0x7f7fe40ffc50 2023-07-12T15:44:26.165 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fea16f700 1 --2- 172.21.15.69:0/1759452589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe41042f0 0x7f7fe41946f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:44:26.165 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fe916d700 1 --2- 172.21.15.69:0/1759452589 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fe40ffc50 0x7f7fe419abc0 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4009fd0 tx=0x7f7fd400ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:44:26.165 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fdaffd700 1 -- 172.21.15.69:0/1759452589 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4013070 con 0x7f7fe40ffc50 2023-07-12T15:44:26.166 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fdaffd700 1 -- 172.21.15.69:0/1759452589 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f7fd400be50 con 0x7f7fe40ffc50 2023-07-12T15:44:26.166 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7febbd2700 1 -- 172.21.15.69:0/1759452589 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe00095f0 con 0x7f7fe40ffc50 2023-07-12T15:44:26.166 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7fdaffd700 1 -- 172.21.15.69:0/1759452589 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4003940 con 0x7f7fe40ffc50 2023-07-12T15:44:26.166 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.159+0000 7f7febbd2700 1 -- 172.21.15.69:0/1759452589 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe4067e30 con 0x7f7fe40ffc50 2023-07-12T15:44:26.166 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.160+0000 7f7febbd2700 1 -- 172.21.15.69:0/1759452589 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe40679e0 con 0x7f7fe40ffc50 2023-07-12T15:44:26.166 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.161+0000 7f7fdaffd700 1 -- 172.21.15.69:0/1759452589 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f7fd400b9c0 con 0x7f7fe40ffc50 2023-07-12T15:44:26.166 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.161+0000 7f7fdaffd700 1 --2- 172.21.15.69:0/1759452589 >> v2:172.21.15.69:6800/1242716914 conn(0x7f7fd006c810 0x7f7fd006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:44:26.167 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.161+0000 7f7fdaffd700 1 -- 172.21.15.69:0/1759452589 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(750..750 src has 1..750) v4 ==== 7671+0+0 (secure 0 0 0) 0x7f7fd408d650 con 0x7f7fe40ffc50 2023-07-12T15:44:26.167 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.161+0000 7f7fe996e700 1 --2- 172.21.15.69:0/1759452589 >> v2:172.21.15.69:6800/1242716914 conn(0x7f7fd006c810 0x7f7fd006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:44:26.167 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.161+0000 7f7fdaffd700 1 -- 172.21.15.69:0/1759452589 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=751}) v3 -- 0x7f7fd00778a0 con 0x7f7fe40ffc50 2023-07-12T15:44:26.167 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.162+0000 7f7fe996e700 1 --2- 172.21.15.69:0/1759452589 >> v2:172.21.15.69:6800/1242716914 conn(0x7f7fd006c810 0x7f7fd006ecc0 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009990 tx=0x7f7fe0009550 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:44:26.167 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.166+0000 7f7fdaffd700 1 -- 172.21.15.69:0/1759452589 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7fd4050720 con 0x7f7fe40ffc50 2023-07-12T15:44:26.360 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:26.359+0000 7f7febbd2700 1 -- 172.21.15.69:0/1759452589 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f7fe4068050 con 0x7f7fe40ffc50 2023-07-12T15:44:26.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:26 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:26 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:26.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:44:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:44:26] "GET /metrics HTTP/1.1" 200 226085 "" "Prometheus/2.43.0" 2023-07-12T15:44:26.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:26 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:27.314 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:27.309+0000 7f7fdaffd700 1 -- 172.21.15.69:0/1759452589 <== mon.2 v2:172.21.15.69:3301/0 7 ==== osd_map(751..751 src has 1..751) v4 ==== 744+0+0 (secure 0 0 0) 0x7f7fd400bc70 con 0x7f7fe40ffc50 2023-07-12T15:44:27.314 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:27.309+0000 7f7fdaffd700 1 -- 172.21.15.69:0/1759452589 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=752}) v3 -- 0x7f7fd0078390 con 0x7f7fe40ffc50 2023-07-12T15:44:27.315 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:27.314+0000 7f7fdaffd700 1 -- 172.21.15.69:0/1759452589 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v751) v1 ==== 225+0+0 (secure 0 0 0) 0x7f7fd405af20 con 0x7f7fe40ffc50 2023-07-12T15:44:27.441 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:27.440+0000 7f7febbd2700 1 -- 172.21.15.69:0/1759452589 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f7fe4068260 con 0x7f7fe40ffc50 2023-07-12T15:44:27.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:27 smithi145 ceph-mon[95801]: pgmap v1171: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:27.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:27 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1759452589' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]: dispatch 2023-07-12T15:44:27.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:27 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]: dispatch 2023-07-12T15:44:27.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:27 smithi069 ceph-mon[102777]: pgmap v1171: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:27.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:27 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1759452589' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]: dispatch 2023-07-12T15:44:27.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:27 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]: dispatch 2023-07-12T15:44:27.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:27 smithi069 ceph-mon[94241]: pgmap v1171: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:27.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:27 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1759452589' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]: dispatch 2023-07-12T15:44:27.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:27 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]: dispatch 2023-07-12T15:44:28.328 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.326+0000 7f7fdaffd700 1 -- 172.21.15.69:0/1759452589 <== mon.2 v2:172.21.15.69:3301/0 9 ==== osd_map(752..752 src has 1..752) v4 ==== 744+0+0 (secure 0 0 0) 0x7f7fd405b150 con 0x7f7fe40ffc50 2023-07-12T15:44:28.328 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.327+0000 7f7fdaffd700 1 -- 172.21.15.69:0/1759452589 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=753}) v3 -- 0x7f7fd0078950 con 0x7f7fe40ffc50 2023-07-12T15:44:28.335 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.331+0000 7f7fdaffd700 1 -- 172.21.15.69:0/1759452589 <== mon.2 v2:172.21.15.69:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v752) v1 ==== 225+0+0 (secure 0 0 0) 0x7f7fd405af20 con 0x7f7fe40ffc50 2023-07-12T15:44:28.337 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.335+0000 7f7febbd2700 1 -- 172.21.15.69:0/1759452589 >> v2:172.21.15.69:6800/1242716914 conn(0x7f7fd006c810 msgr2=0x7f7fd006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:44:28.337 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.335+0000 7f7febbd2700 1 --2- 172.21.15.69:0/1759452589 >> v2:172.21.15.69:6800/1242716914 conn(0x7f7fd006c810 0x7f7fd006ecc0 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009990 tx=0x7f7fe0009550 comp rx=0 tx=0).stop 2023-07-12T15:44:28.337 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.335+0000 7f7febbd2700 1 -- 172.21.15.69:0/1759452589 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fe40ffc50 msgr2=0x7f7fe419abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:44:28.337 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.335+0000 7f7febbd2700 1 --2- 172.21.15.69:0/1759452589 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fe40ffc50 0x7f7fe419abc0 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4009fd0 tx=0x7f7fd400ee20 comp rx=0 tx=0).stop 2023-07-12T15:44:28.338 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.335+0000 7f7febbd2700 1 -- 172.21.15.69:0/1759452589 shutdown_connections 2023-07-12T15:44:28.338 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.335+0000 7f7febbd2700 1 --2- 172.21.15.69:0/1759452589 >> v2:172.21.15.69:6800/1242716914 conn(0x7f7fd006c810 0x7f7fd006ecc0 unknown :-1 s=CLOSED pgs=3226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:28.338 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.335+0000 7f7febbd2700 1 --2- 172.21.15.69:0/1759452589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe41042f0 0x7f7fe41946f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:28.338 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.335+0000 7f7febbd2700 1 --2- 172.21.15.69:0/1759452589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7fe40ff2b0 0x7f7fe419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:28.338 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.335+0000 7f7febbd2700 1 --2- 172.21.15.69:0/1759452589 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fe40ffc50 0x7f7fe419abc0 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:28.338 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.335+0000 7f7febbd2700 1 -- 172.21.15.69:0/1759452589 >> 172.21.15.69:0/1759452589 conn(0x7f7fe4075e10 msgr2=0x7f7fe40fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:44:28.339 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.335+0000 7f7febbd2700 1 -- 172.21.15.69:0/1759452589 shutdown_connections 2023-07-12T15:44:28.339 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:28.335+0000 7f7febbd2700 1 -- 172.21.15.69:0/1759452589 wait complete. 2023-07-12T15:44:28.339 INFO:tasks.workunit.client.0.smithi069.stderr:set-quota max_objects = 100 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 2023-07-12T15:44:28.349 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 204153 2023-07-12T15:44:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:28 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]': finished 2023-07-12T15:44:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:28 smithi145 ceph-mon[95801]: osdmap e751: 8 total, 8 up, 8 in 2023-07-12T15:44:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:28 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1759452589' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]: dispatch 2023-07-12T15:44:28.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:28 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]: dispatch 2023-07-12T15:44:28.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:28 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]': finished 2023-07-12T15:44:28.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:28 smithi069 ceph-mon[102777]: osdmap e751: 8 total, 8 up, 8 in 2023-07-12T15:44:28.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:28 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1759452589' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]: dispatch 2023-07-12T15:44:28.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:28 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]: dispatch 2023-07-12T15:44:28.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:28 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]': finished 2023-07-12T15:44:28.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:28 smithi069 ceph-mon[94241]: osdmap e751: 8 total, 8 up, 8 in 2023-07-12T15:44:28.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:28 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1759452589' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]: dispatch 2023-07-12T15:44:28.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:28 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]: dispatch 2023-07-12T15:44:29.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:29 smithi145 ceph-mon[95801]: pgmap v1173: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:44:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:29 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]': finished 2023-07-12T15:44:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:29 smithi145 ceph-mon[95801]: osdmap e752: 8 total, 8 up, 8 in 2023-07-12T15:44:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:44:29.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:29 smithi069 ceph-mon[102777]: pgmap v1173: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:44:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:29 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]': finished 2023-07-12T15:44:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:29 smithi069 ceph-mon[102777]: osdmap e752: 8 total, 8 up, 8 in 2023-07-12T15:44:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:44:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:29 smithi069 ceph-mon[94241]: pgmap v1173: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:44:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:29 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "100"}]': finished 2023-07-12T15:44:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:29 smithi069 ceph-mon[94241]: osdmap e752: 8 total, 8 up, 8 in 2023-07-12T15:44:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:44:31.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:31 smithi145 ceph-mon[95801]: pgmap v1175: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:31.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:31 smithi069 ceph-mon[102777]: pgmap v1175: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:31.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:31 smithi069 ceph-mon[94241]: pgmap v1175: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:31.968 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 0 -ne 0 ']' 2023-07-12T15:44:31.969 INFO:tasks.workunit.client.0.smithi069.stderr:+ true 2023-07-12T15:44:31.969 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put twomore /etc/passwd 2023-07-12T15:44:32.006 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool set-quota e6659867-2dff-445e-9a1e-87d1f41b1a67 max_bytes 100 2023-07-12T15:44:32.110 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.109+0000 7f1ca31a0700 1 -- 172.21.15.69:0/1533929578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c9c0ffd20 msgr2=0x7f1c9c103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:44:32.111 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.109+0000 7f1ca31a0700 1 --2- 172.21.15.69:0/1533929578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c9c0ffd20 0x7f1c9c103260 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009990 tx=0x7f1c98009ca0 comp rx=0 tx=0).stop 2023-07-12T15:44:32.111 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.109+0000 7f1ca31a0700 1 -- 172.21.15.69:0/1533929578 shutdown_connections 2023-07-12T15:44:32.111 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.109+0000 7f1ca31a0700 1 --2- 172.21.15.69:0/1533929578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c9c103990 0x7f1c9c10b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:32.111 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.109+0000 7f1ca31a0700 1 --2- 172.21.15.69:0/1533929578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c9c0ff380 0x7f1c9c0ff750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:32.111 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.109+0000 7f1ca31a0700 1 --2- 172.21.15.69:0/1533929578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c9c0ffd20 0x7f1c9c103260 unknown :-1 s=CLOSED pgs=2773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:32.111 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.109+0000 7f1ca31a0700 1 -- 172.21.15.69:0/1533929578 >> 172.21.15.69:0/1533929578 conn(0x7f1c9c074700 msgr2=0x7f1c9c074b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:44:32.111 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.109+0000 7f1ca31a0700 1 -- 172.21.15.69:0/1533929578 shutdown_connections 2023-07-12T15:44:32.112 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.109+0000 7f1ca31a0700 1 -- 172.21.15.69:0/1533929578 wait complete. 2023-07-12T15:44:32.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca31a0700 1 Processor -- start 2023-07-12T15:44:32.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca31a0700 1 -- start start 2023-07-12T15:44:32.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca31a0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c9c0ff380 0x7f1c9c195680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:44:32.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca31a0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c9c0ffd20 0x7f1c9c195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:44:32.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca31a0700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c9c103990 0x7f1c9c19a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:44:32.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca31a0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c19a660 con 0x7f1c9c0ff380 2023-07-12T15:44:32.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca31a0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c19a7d0 con 0x7f1c9c0ffd20 2023-07-12T15:44:32.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca31a0700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c19a990 con 0x7f1c9c103990 2023-07-12T15:44:32.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca219e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c9c0ff380 0x7f1c9c195680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:44:32.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca219e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c9c0ff380 0x7f1c9c195680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47842/0 (socket says 172.21.15.69:47842) 2023-07-12T15:44:32.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca219e700 1 -- 172.21.15.69:0/2852292448 learned_addr learned my addr 172.21.15.69:0/2852292448 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:44:32.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca299f700 1 --2- 172.21.15.69:0/2852292448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c9c103990 0x7f1c9c19a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:44:32.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca199d700 1 --2- 172.21.15.69:0/2852292448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c9c0ffd20 0x7f1c9c195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:44:32.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca219e700 1 -- 172.21.15.69:0/2852292448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c9c103990 msgr2=0x7f1c9c19a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:44:32.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca219e700 1 --2- 172.21.15.69:0/2852292448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c9c103990 0x7f1c9c19a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:32.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca219e700 1 -- 172.21.15.69:0/2852292448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c9c0ffd20 msgr2=0x7f1c9c195bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:44:32.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca219e700 1 --2- 172.21.15.69:0/2852292448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c9c0ffd20 0x7f1c9c195bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:32.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca219e700 1 -- 172.21.15.69:0/2852292448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c009520 con 0x7f1c9c0ff380 2023-07-12T15:44:32.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca299f700 1 --2- 172.21.15.69:0/2852292448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c9c103990 0x7f1c9c19a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:44:32.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca219e700 1 --2- 172.21.15.69:0/2852292448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c9c0ff380 0x7f1c9c195680 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c00eb30 tx=0x7f1c8c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:44:32.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1c937fe700 1 -- 172.21.15.69:0/2852292448 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c009880 con 0x7f1c9c0ff380 2023-07-12T15:44:32.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1c937fe700 1 -- 172.21.15.69:0/2852292448 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f1c8c0044b0 con 0x7f1c9c0ff380 2023-07-12T15:44:32.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1ca31a0700 1 -- 172.21.15.69:0/2852292448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c980095f0 con 0x7f1c9c0ff380 2023-07-12T15:44:32.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.110+0000 7f1c937fe700 1 -- 172.21.15.69:0/2852292448 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c003d80 con 0x7f1c9c0ff380 2023-07-12T15:44:32.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.111+0000 7f1ca31a0700 1 -- 172.21.15.69:0/2852292448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c9c1a22d0 con 0x7f1c9c0ff380 2023-07-12T15:44:32.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.111+0000 7f1ca31a0700 1 -- 172.21.15.69:0/2852292448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c9c04e800 con 0x7f1c9c0ff380 2023-07-12T15:44:32.119 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.112+0000 7f1c937fe700 1 -- 172.21.15.69:0/2852292448 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f1c8c003ee0 con 0x7f1c9c0ff380 2023-07-12T15:44:32.119 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.112+0000 7f1c937fe700 1 --2- 172.21.15.69:0/2852292448 >> v2:172.21.15.69:6800/1242716914 conn(0x7f1c8806c810 0x7f1c8806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:44:32.119 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.112+0000 7f1c937fe700 1 -- 172.21.15.69:0/2852292448 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(753..753 src has 1..753) v4 ==== 7671+0+0 (secure 0 0 0) 0x7f1c8c014070 con 0x7f1c9c0ff380 2023-07-12T15:44:32.119 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.112+0000 7f1ca199d700 1 --2- 172.21.15.69:0/2852292448 >> v2:172.21.15.69:6800/1242716914 conn(0x7f1c8806c810 0x7f1c8806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:44:32.119 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.113+0000 7f1ca199d700 1 --2- 172.21.15.69:0/2852292448 >> v2:172.21.15.69:6800/1242716914 conn(0x7f1c8806c810 0x7f1c8806ecc0 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800b6f0 tx=0x7f1c98005fb0 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:44:32.120 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.117+0000 7f1c937fe700 1 -- 172.21.15.69:0/2852292448 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1c8c059f70 con 0x7f1c9c0ff380 2023-07-12T15:44:32.315 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.314+0000 7f1ca31a0700 1 -- 172.21.15.69:0/2852292448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f1c9c1a2520 con 0x7f1c9c0ff380 2023-07-12T15:44:32.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:32 smithi145 ceph-mon[95801]: pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' no longer out of quota; removing NO_QUOTA flag 2023-07-12T15:44:32.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:32 smithi145 ceph-mon[95801]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2023-07-12T15:44:32.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:32 smithi145 ceph-mon[95801]: osdmap e753: 8 total, 8 up, 8 in 2023-07-12T15:44:32.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:32 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2852292448' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]: dispatch 2023-07-12T15:44:32.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:32 smithi069 ceph-mon[102777]: pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' no longer out of quota; removing NO_QUOTA flag 2023-07-12T15:44:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:32 smithi069 ceph-mon[102777]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2023-07-12T15:44:32.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:32 smithi069 ceph-mon[102777]: osdmap e753: 8 total, 8 up, 8 in 2023-07-12T15:44:32.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:32 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2852292448' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]: dispatch 2023-07-12T15:44:32.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:32 smithi069 ceph-mon[94241]: pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' no longer out of quota; removing NO_QUOTA flag 2023-07-12T15:44:32.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:32 smithi069 ceph-mon[94241]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2023-07-12T15:44:32.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:32 smithi069 ceph-mon[94241]: osdmap e753: 8 total, 8 up, 8 in 2023-07-12T15:44:32.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:32 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2852292448' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]: dispatch 2023-07-12T15:44:32.978 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:32.977+0000 7f1c937fe700 1 -- 172.21.15.69:0/2852292448 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v754) v1 ==== 221+0+0 (secure 0 0 0) 0x7f1c8c059b00 con 0x7f1c9c0ff380 2023-07-12T15:44:33.109 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.108+0000 7f1ca31a0700 1 -- 172.21.15.69:0/2852292448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f1c9c1969d0 con 0x7f1c9c0ff380 2023-07-12T15:44:33.600 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:44:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a[122798]: logger=sqlstore.transactions t=2023-07-12T15:44:33.138994541Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2023-07-12T15:44:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:33 smithi145 ceph-mon[95801]: pgmap v1177: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:44:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:33 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:44:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:33 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:33 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:33 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:33 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2852292448' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]': finished 2023-07-12T15:44:33.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:33 smithi145 ceph-mon[95801]: osdmap e754: 8 total, 8 up, 8 in 2023-07-12T15:44:33.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:33 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2852292448' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]: dispatch 2023-07-12T15:44:33.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:33 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:44:33.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:33 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:44:33.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:33 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[102777]: pgmap v1177: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:44:33.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:44:33.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2852292448' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]': finished 2023-07-12T15:44:33.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[102777]: osdmap e754: 8 total, 8 up, 8 in 2023-07-12T15:44:33.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2852292448' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]: dispatch 2023-07-12T15:44:33.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:44:33.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:44:33.737 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[94241]: pgmap v1177: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:44:33.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:44:33.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2852292448' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]': finished 2023-07-12T15:44:33.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[94241]: osdmap e754: 8 total, 8 up, 8 in 2023-07-12T15:44:33.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2852292448' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]: dispatch 2023-07-12T15:44:33.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:44:33.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:44:33.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:33 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:44:33.978 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.977+0000 7f1c937fe700 1 -- 172.21.15.69:0/2852292448 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v755) v1 ==== 221+0+0 (secure 0 0 0) 0x7f1c8c059ce0 con 0x7f1c9c0ff380 2023-07-12T15:44:33.981 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.980+0000 7f1ca31a0700 1 -- 172.21.15.69:0/2852292448 >> v2:172.21.15.69:6800/1242716914 conn(0x7f1c8806c810 msgr2=0x7f1c8806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:44:33.981 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.980+0000 7f1ca31a0700 1 --2- 172.21.15.69:0/2852292448 >> v2:172.21.15.69:6800/1242716914 conn(0x7f1c8806c810 0x7f1c8806ecc0 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800b6f0 tx=0x7f1c98005fb0 comp rx=0 tx=0).stop 2023-07-12T15:44:33.982 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.980+0000 7f1ca31a0700 1 -- 172.21.15.69:0/2852292448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c9c0ff380 msgr2=0x7f1c9c195680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:44:33.982 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.980+0000 7f1ca31a0700 1 --2- 172.21.15.69:0/2852292448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c9c0ff380 0x7f1c9c195680 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c00eb30 tx=0x7f1c8c00ee40 comp rx=0 tx=0).stop 2023-07-12T15:44:33.982 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.980+0000 7f1ca31a0700 1 -- 172.21.15.69:0/2852292448 shutdown_connections 2023-07-12T15:44:33.982 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.980+0000 7f1ca31a0700 1 --2- 172.21.15.69:0/2852292448 >> v2:172.21.15.69:6800/1242716914 conn(0x7f1c8806c810 0x7f1c8806ecc0 unknown :-1 s=CLOSED pgs=3228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:33.982 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.980+0000 7f1ca31a0700 1 --2- 172.21.15.69:0/2852292448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c9c103990 0x7f1c9c19a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:33.982 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.980+0000 7f1ca31a0700 1 --2- 172.21.15.69:0/2852292448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c9c0ff380 0x7f1c9c195680 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:33.983 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.980+0000 7f1ca31a0700 1 --2- 172.21.15.69:0/2852292448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c9c0ffd20 0x7f1c9c195bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:44:33.983 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.980+0000 7f1ca31a0700 1 -- 172.21.15.69:0/2852292448 >> 172.21.15.69:0/2852292448 conn(0x7f1c9c074700 msgr2=0x7f1c9c10bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:44:33.983 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.980+0000 7f1ca31a0700 1 -- 172.21.15.69:0/2852292448 shutdown_connections 2023-07-12T15:44:33.983 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:44:33.980+0000 7f1ca31a0700 1 -- 172.21.15.69:0/2852292448 wait complete. 2023-07-12T15:44:33.983 INFO:tasks.workunit.client.0.smithi069.stderr:set-quota max_bytes = 100 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 2023-07-12T15:44:33.994 INFO:tasks.workunit.client.0.smithi069.stderr:+ sleep 30 2023-07-12T15:44:35.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:34 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2852292448' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]': finished 2023-07-12T15:44:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:34 smithi069 ceph-mon[102777]: osdmap e755: 8 total, 8 up, 8 in 2023-07-12T15:44:35.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:34 smithi069 ceph-mon[102777]: pgmap v1180: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2023-07-12T15:44:35.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:34 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2852292448' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]': finished 2023-07-12T15:44:35.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:34 smithi069 ceph-mon[94241]: osdmap e755: 8 total, 8 up, 8 in 2023-07-12T15:44:35.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:34 smithi069 ceph-mon[94241]: pgmap v1180: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2023-07-12T15:44:35.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:34 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2852292448' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "100"}]': finished 2023-07-12T15:44:35.314 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:34 smithi145 ceph-mon[95801]: osdmap e755: 8 total, 8 up, 8 in 2023-07-12T15:44:35.314 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:34 smithi145 ceph-mon[95801]: pgmap v1180: 176 pgs: 176 active+clean; 689 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2023-07-12T15:44:35.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:44:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:44:36.235 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:44:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:44:36] "GET /metrics HTTP/1.1" 200 226028 "" "Prometheus/2.43.0" 2023-07-12T15:44:36.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:35 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:36.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:35 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:36.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:35 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:37.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:36 smithi145 ceph-mon[95801]: pgmap v1181: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 3.7 KiB/s wr, 1 op/s 2023-07-12T15:44:37.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:36 smithi145 ceph-mon[95801]: pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' is full (reached quota's max_bytes: 100 B) 2023-07-12T15:44:37.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:36 smithi145 ceph-mon[95801]: Health check failed: 1 pool(s) full (POOL_FULL) 2023-07-12T15:44:37.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:36 smithi145 ceph-mon[95801]: osdmap e756: 8 total, 8 up, 8 in 2023-07-12T15:44:37.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:36 smithi069 ceph-mon[102777]: pgmap v1181: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 3.7 KiB/s wr, 1 op/s 2023-07-12T15:44:37.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:36 smithi069 ceph-mon[102777]: pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' is full (reached quota's max_bytes: 100 B) 2023-07-12T15:44:37.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:36 smithi069 ceph-mon[102777]: Health check failed: 1 pool(s) full (POOL_FULL) 2023-07-12T15:44:37.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:36 smithi069 ceph-mon[102777]: osdmap e756: 8 total, 8 up, 8 in 2023-07-12T15:44:37.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:36 smithi069 ceph-mon[94241]: pgmap v1181: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 3.7 KiB/s wr, 1 op/s 2023-07-12T15:44:37.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:36 smithi069 ceph-mon[94241]: pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' is full (reached quota's max_bytes: 100 B) 2023-07-12T15:44:37.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:36 smithi069 ceph-mon[94241]: Health check failed: 1 pool(s) full (POOL_FULL) 2023-07-12T15:44:37.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:36 smithi069 ceph-mon[94241]: osdmap e756: 8 total, 8 up, 8 in 2023-07-12T15:44:39.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:39 smithi145 ceph-mon[95801]: pgmap v1183: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 3.7 KiB/s wr, 1 op/s 2023-07-12T15:44:39.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:39 smithi069 ceph-mon[102777]: pgmap v1183: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 3.7 KiB/s wr, 1 op/s 2023-07-12T15:44:39.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:39 smithi069 ceph-mon[94241]: pgmap v1183: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 3.7 KiB/s wr, 1 op/s 2023-07-12T15:44:41.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:41 smithi145 ceph-mon[95801]: pgmap v1184: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2023-07-12T15:44:41.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:41 smithi069 ceph-mon[102777]: pgmap v1184: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2023-07-12T15:44:41.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:41 smithi069 ceph-mon[94241]: pgmap v1184: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2023-07-12T15:44:43.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:42 smithi069 ceph-mon[102777]: pgmap v1185: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2023-07-12T15:44:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:42 smithi069 ceph-mon[94241]: pgmap v1185: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2023-07-12T15:44:43.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:42 smithi145 ceph-mon[95801]: pgmap v1185: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2023-07-12T15:44:44.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:43 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:44:44.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:43 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:44:44.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:43 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:44:45.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:44 smithi069 ceph-mon[94241]: pgmap v1186: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 1 op/s 2023-07-12T15:44:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:44 smithi069 ceph-mon[102777]: pgmap v1186: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 1 op/s 2023-07-12T15:44:45.325 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:44 smithi145 ceph-mon[95801]: pgmap v1186: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 1 op/s 2023-07-12T15:44:45.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:44:45 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:44:46.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:45 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:46.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:45 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:46.236 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:44:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:44:46] "GET /metrics HTTP/1.1" 200 226053 "" "Prometheus/2.43.0" 2023-07-12T15:44:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:45 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:47.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:46 smithi069 ceph-mon[102777]: pgmap v1187: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:44:47.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:46 smithi069 ceph-mon[94241]: pgmap v1187: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:44:47.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:46 smithi145 ceph-mon[95801]: pgmap v1187: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:44:49.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:49 smithi145 ceph-mon[95801]: pgmap v1188: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 906 B/s rd, 0 op/s 2023-07-12T15:44:49.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:49 smithi069 ceph-mon[102777]: pgmap v1188: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 906 B/s rd, 0 op/s 2023-07-12T15:44:49.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:49 smithi069 ceph-mon[94241]: pgmap v1188: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 906 B/s rd, 0 op/s 2023-07-12T15:44:51.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:51 smithi145 ceph-mon[95801]: pgmap v1189: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:51.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:51 smithi069 ceph-mon[102777]: pgmap v1189: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:51.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:51 smithi069 ceph-mon[94241]: pgmap v1189: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:53.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:52 smithi069 ceph-mon[102777]: pgmap v1190: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:53.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:52 smithi069 ceph-mon[94241]: pgmap v1190: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:53.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:52 smithi145 ceph-mon[95801]: pgmap v1190: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:55.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:44:55 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:44:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:55 smithi145 ceph-mon[95801]: pgmap v1191: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:55.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:55 smithi069 ceph-mon[102777]: pgmap v1191: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:55.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:55 smithi069 ceph-mon[94241]: pgmap v1191: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:56.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:56 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:56.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:56 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:56.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:44:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:44:56] "GET /metrics HTTP/1.1" 200 226053 "" "Prometheus/2.43.0" 2023-07-12T15:44:56.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:56 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:44:57.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:57 smithi145 ceph-mon[95801]: pgmap v1192: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:57.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:57 smithi069 ceph-mon[102777]: pgmap v1192: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:57.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:57 smithi069 ceph-mon[94241]: pgmap v1192: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:44:59.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:59 smithi145 ceph-mon[95801]: pgmap v1193: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:59.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:44:59 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:44:59.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:59 smithi069 ceph-mon[102777]: pgmap v1193: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:59.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:44:59 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:44:59.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:59 smithi069 ceph-mon[94241]: pgmap v1193: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:44:59.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:44:59 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:45:01.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:01 smithi145 ceph-mon[95801]: pgmap v1194: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:01.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:01 smithi069 ceph-mon[102777]: pgmap v1194: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:01.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:01 smithi069 ceph-mon[94241]: pgmap v1194: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:03.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:02 smithi069 ceph-mon[102777]: pgmap v1195: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:02 smithi069 ceph-mon[94241]: pgmap v1195: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:03.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:02 smithi145 ceph-mon[95801]: pgmap v1195: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:03.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=204452 2023-07-12T15:45:03.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool set-quota e6659867-2dff-445e-9a1e-87d1f41b1a67 max_bytes 0 2023-07-12T15:45:03.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put two /etc/passwd 2023-07-12T15:45:04.112 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.109+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2314570740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30100220 msgr2=0x7fcf301005f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:04.112 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.109+0000 7fcf35fa1700 1 --2- 172.21.15.69:0/2314570740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30100220 0x7fcf301005f0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800b0d0 tx=0x7fcf2800b3e0 comp rx=0 tx=0).stop 2023-07-12T15:45:04.112 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.109+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2314570740 shutdown_connections 2023-07-12T15:45:04.113 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.109+0000 7fcf35fa1700 1 --2- 172.21.15.69:0/2314570740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf301097d0 0x7fcf3010bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:04.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.109+0000 7fcf35fa1700 1 --2- 172.21.15.69:0/2314570740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30100220 0x7fcf301005f0 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:04.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.109+0000 7fcf35fa1700 1 --2- 172.21.15.69:0/2314570740 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf30100b30 0x7fcf30109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:04.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.109+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2314570740 >> 172.21.15.69:0/2314570740 conn(0x7fcf30074700 msgr2=0x7fcf30074b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:45:04.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.109+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2314570740 shutdown_connections 2023-07-12T15:45:04.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.109+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2314570740 wait complete. 2023-07-12T15:45:04.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf35fa1700 1 Processor -- start 2023-07-12T15:45:04.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf35fa1700 1 -- start start 2023-07-12T15:45:04.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf35fa1700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf30100220 0x7fcf30196730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:04.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf35fa1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30100b30 0x7fcf30196c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:04.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf35fa1700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf301097d0 0x7fcf301a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:04.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf35fa1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf301975b0 con 0x7fcf30100b30 2023-07-12T15:45:04.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf35fa1700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf301a1cf0 con 0x7fcf30100220 2023-07-12T15:45:04.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf35fa1700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf301a1eb0 con 0x7fcf301097d0 2023-07-12T15:45:04.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf357a0700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf301097d0 0x7fcf301a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:04.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf2ffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30100b30 0x7fcf30196c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:04.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf357a0700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf301097d0 0x7fcf301a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41312/0 (socket says 172.21.15.69:41312) 2023-07-12T15:45:04.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf357a0700 1 -- 172.21.15.69:0/2544431538 learned_addr learned my addr 172.21.15.69:0/2544431538 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:45:04.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf2ffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30100b30 0x7fcf30196c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47882/0 (socket says 172.21.15.69:47882) 2023-07-12T15:45:04.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf34f9f700 1 --2- 172.21.15.69:0/2544431538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf30100220 0x7fcf30196730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:04.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf2ffff700 1 -- 172.21.15.69:0/2544431538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf301097d0 msgr2=0x7fcf301a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:04.119 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf2ffff700 1 --2- 172.21.15.69:0/2544431538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf301097d0 0x7fcf301a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:04.119 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf2ffff700 1 -- 172.21.15.69:0/2544431538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf30100220 msgr2=0x7fcf30196730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:04.119 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf2ffff700 1 --2- 172.21.15.69:0/2544431538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf30100220 0x7fcf30196730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:04.119 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf2ffff700 1 -- 172.21.15.69:0/2544431538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf28009d00 con 0x7fcf30100b30 2023-07-12T15:45:04.119 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf357a0700 1 --2- 172.21.15.69:0/2544431538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf301097d0 0x7fcf301a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:45:04.119 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.110+0000 7fcf2ffff700 1 --2- 172.21.15.69:0/2544431538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30100b30 0x7fcf30196c70 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009c50 tx=0x7fcf18009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:45:04.119 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.111+0000 7fcf2dffb700 1 -- 172.21.15.69:0/2544431538 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18015070 con 0x7fcf30100b30 2023-07-12T15:45:04.120 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.111+0000 7fcf2dffb700 1 -- 172.21.15.69:0/2544431538 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fcf18019410 con 0x7fcf30100b30 2023-07-12T15:45:04.120 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.111+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2544431538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf30194580 con 0x7fcf30100b30 2023-07-12T15:45:04.120 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.111+0000 7fcf2dffb700 1 -- 172.21.15.69:0/2544431538 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18018610 con 0x7fcf30100b30 2023-07-12T15:45:04.120 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.111+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2544431538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf30194a90 con 0x7fcf30100b30 2023-07-12T15:45:04.120 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.112+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2544431538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3004e800 con 0x7fcf30100b30 2023-07-12T15:45:04.120 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.112+0000 7fcf2dffb700 1 -- 172.21.15.69:0/2544431538 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7fcf1800ba50 con 0x7fcf30100b30 2023-07-12T15:45:04.120 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.113+0000 7fcf2dffb700 1 --2- 172.21.15.69:0/2544431538 >> v2:172.21.15.69:6800/1242716914 conn(0x7fcf1c06c810 0x7fcf1c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:04.121 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.113+0000 7fcf2dffb700 1 -- 172.21.15.69:0/2544431538 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(756..756 src has 1..756) v4 ==== 7671+0+0 (secure 0 0 0) 0x7fcf1808f7e0 con 0x7fcf30100b30 2023-07-12T15:45:04.121 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.113+0000 7fcf34f9f700 1 --2- 172.21.15.69:0/2544431538 >> v2:172.21.15.69:6800/1242716914 conn(0x7fcf1c06c810 0x7fcf1c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:04.121 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.113+0000 7fcf2dffb700 1 -- 172.21.15.69:0/2544431538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=757}) v3 -- 0x7fcf1c077820 con 0x7fcf30100b30 2023-07-12T15:45:04.121 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.113+0000 7fcf34f9f700 1 --2- 172.21.15.69:0/2544431538 >> v2:172.21.15.69:6800/1242716914 conn(0x7fcf1c06c810 0x7fcf1c06ecc0 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 crypto rx=0x7fcf28007ad0 tx=0x7fcf2800b040 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:45:04.121 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.117+0000 7fcf2dffb700 1 -- 172.21.15.69:0/2544431538 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcf18053720 con 0x7fcf30100b30 2023-07-12T15:45:04.311 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.309+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2544431538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7fcf30194d10 con 0x7fcf30100b30 2023-07-12T15:45:04.364 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.362+0000 7fcf2dffb700 1 -- 172.21.15.69:0/2544431538 <== mon.0 v2:172.21.15.69:3300/0 7 ==== osd_map(757..757 src has 1..757) v4 ==== 744+0+0 (secure 0 0 0) 0x7fcf1805d390 con 0x7fcf30100b30 2023-07-12T15:45:04.365 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.362+0000 7fcf2dffb700 1 -- 172.21.15.69:0/2544431538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=758}) v3 -- 0x7fcf1c078370 con 0x7fcf30100b30 2023-07-12T15:45:04.368 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.365+0000 7fcf2dffb700 1 -- 172.21.15.69:0/2544431538 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v757) v1 ==== 217+0+0 (secure 0 0 0) 0x7fcf1801e0d0 con 0x7fcf30100b30 2023-07-12T15:45:04.495 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:04.493+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2544431538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7fcf30194f20 con 0x7fcf30100b30 2023-07-12T15:45:04.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:04 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2544431538' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:45:04.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:04 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2544431538' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:45:04.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:04 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2544431538' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:45:05.374 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.371+0000 7fcf2dffb700 1 -- 172.21.15.69:0/2544431538 <== mon.0 v2:172.21.15.69:3300/0 9 ==== osd_map(758..758 src has 1..758) v4 ==== 744+0+0 (secure 0 0 0) 0x7fcf18061310 con 0x7fcf30100b30 2023-07-12T15:45:05.374 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.371+0000 7fcf2dffb700 1 -- 172.21.15.69:0/2544431538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=759}) v3 -- 0x7fcf1c078930 con 0x7fcf30100b30 2023-07-12T15:45:05.379 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.377+0000 7fcf2dffb700 1 -- 172.21.15.69:0/2544431538 <== mon.0 v2:172.21.15.69:3300/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v758) v1 ==== 217+0+0 (secure 0 0 0) 0x7fcf1805d160 con 0x7fcf30100b30 2023-07-12T15:45:05.383 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.381+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2544431538 >> v2:172.21.15.69:6800/1242716914 conn(0x7fcf1c06c810 msgr2=0x7fcf1c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:05.384 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.381+0000 7fcf35fa1700 1 --2- 172.21.15.69:0/2544431538 >> v2:172.21.15.69:6800/1242716914 conn(0x7fcf1c06c810 0x7fcf1c06ecc0 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 crypto rx=0x7fcf28007ad0 tx=0x7fcf2800b040 comp rx=0 tx=0).stop 2023-07-12T15:45:05.384 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.381+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2544431538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30100b30 msgr2=0x7fcf30196c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:05.384 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.381+0000 7fcf35fa1700 1 --2- 172.21.15.69:0/2544431538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30100b30 0x7fcf30196c70 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009c50 tx=0x7fcf18009f60 comp rx=0 tx=0).stop 2023-07-12T15:45:05.384 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.381+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2544431538 shutdown_connections 2023-07-12T15:45:05.384 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.381+0000 7fcf35fa1700 1 --2- 172.21.15.69:0/2544431538 >> v2:172.21.15.69:6800/1242716914 conn(0x7fcf1c06c810 0x7fcf1c06ecc0 unknown :-1 s=CLOSED pgs=3230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:05.384 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.381+0000 7fcf35fa1700 1 --2- 172.21.15.69:0/2544431538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf301097d0 0x7fcf301a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:05.385 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.381+0000 7fcf35fa1700 1 --2- 172.21.15.69:0/2544431538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf30100220 0x7fcf30196730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:05.385 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.381+0000 7fcf35fa1700 1 --2- 172.21.15.69:0/2544431538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30100b30 0x7fcf30196c70 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:05.385 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.381+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2544431538 >> 172.21.15.69:0/2544431538 conn(0x7fcf30074700 msgr2=0x7fcf300fc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:45:05.385 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.381+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2544431538 shutdown_connections 2023-07-12T15:45:05.385 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.381+0000 7fcf35fa1700 1 -- 172.21.15.69:0/2544431538 wait complete. 2023-07-12T15:45:05.386 INFO:tasks.workunit.client.0.smithi069.stderr:set-quota max_bytes = 0 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 2023-07-12T15:45:05.396 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool set-quota e6659867-2dff-445e-9a1e-87d1f41b1a67 max_objects 0 2023-07-12T15:45:05.502 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.499+0000 7f05a5e0b700 1 -- 172.21.15.69:0/1782823243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05a00ffc80 msgr2=0x7f05a010b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:05.502 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.499+0000 7f05a5e0b700 1 --2- 172.21.15.69:0/1782823243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05a00ffc80 0x7f05a010b5f0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f059800b0a0 tx=0x7f059800b3b0 comp rx=0 tx=0).stop 2023-07-12T15:45:05.503 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.499+0000 7f05a5e0b700 1 -- 172.21.15.69:0/1782823243 shutdown_connections 2023-07-12T15:45:05.503 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.499+0000 7f05a5e0b700 1 --2- 172.21.15.69:0/1782823243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05a00ffc80 0x7f05a010b5f0 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:05.503 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.499+0000 7f05a5e0b700 1 --2- 172.21.15.69:0/1782823243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05a0104930 0x7f05a0104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:05.504 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.499+0000 7f05a5e0b700 1 --2- 172.21.15.69:0/1782823243 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05a00ff2d0 0x7f05a00ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:05.504 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.499+0000 7f05a5e0b700 1 -- 172.21.15.69:0/1782823243 >> 172.21.15.69:0/1782823243 conn(0x7f05a00748c0 msgr2=0x7f05a0074cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:45:05.504 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.499+0000 7f05a5e0b700 1 -- 172.21.15.69:0/1782823243 shutdown_connections 2023-07-12T15:45:05.504 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.499+0000 7f05a5e0b700 1 -- 172.21.15.69:0/1782823243 wait complete. 2023-07-12T15:45:05.505 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.500+0000 7f05a5e0b700 1 Processor -- start 2023-07-12T15:45:05.508 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.500+0000 7f05a5e0b700 1 -- start start 2023-07-12T15:45:05.509 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.500+0000 7f05a5e0b700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05a00ff2d0 0x7f05a01949b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:05.509 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.500+0000 7f05a5e0b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05a00ffc80 0x7f05a0194ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:05.509 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.500+0000 7f05a5e0b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05a0104930 0x7f05a0199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:05.509 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.500+0000 7f05a5e0b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a0199950 con 0x7f05a00ffc80 2023-07-12T15:45:05.510 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.500+0000 7f05a5e0b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a0199ac0 con 0x7f05a0104930 2023-07-12T15:45:05.510 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.500+0000 7f05a5e0b700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a0199c80 con 0x7f05a00ff2d0 2023-07-12T15:45:05.510 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059f7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05a00ff2d0 0x7f05a01949b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:05.510 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059f7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05a00ff2d0 0x7f05a01949b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41322/0 (socket says 172.21.15.69:41322) 2023-07-12T15:45:05.510 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059effd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05a00ffc80 0x7f05a0194ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:05.510 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059f7fe700 1 -- 172.21.15.69:0/3601315157 learned_addr learned my addr 172.21.15.69:0/3601315157 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:45:05.511 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059f7fe700 1 -- 172.21.15.69:0/3601315157 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05a0104930 msgr2=0x7f05a0199380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:05.511 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059f7fe700 1 --2- 172.21.15.69:0/3601315157 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05a0104930 0x7f05a0199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:05.511 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059f7fe700 1 -- 172.21.15.69:0/3601315157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05a00ffc80 msgr2=0x7f05a0194ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:05.511 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059f7fe700 1 --2- 172.21.15.69:0/3601315157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05a00ffc80 0x7f05a0194ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:05.511 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059f7fe700 1 -- 172.21.15.69:0/3601315157 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0588009520 con 0x7f05a00ff2d0 2023-07-12T15:45:05.511 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059effd700 1 --2- 172.21.15.69:0/3601315157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05a00ffc80 0x7f05a0194ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:45:05.511 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059f7fe700 1 --2- 172.21.15.69:0/3601315157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05a00ff2d0 0x7f05a01949b0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7f058800eb30 tx=0x7f058800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:45:05.512 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059cff9700 1 -- 172.21.15.69:0/3601315157 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588009880 con 0x7f05a00ff2d0 2023-07-12T15:45:05.512 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059cff9700 1 -- 172.21.15.69:0/3601315157 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f05880044b0 con 0x7f05a00ff2d0 2023-07-12T15:45:05.512 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f05a5e0b700 1 -- 172.21.15.69:0/3601315157 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0598009d00 con 0x7f05a00ff2d0 2023-07-12T15:45:05.512 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f059cff9700 1 -- 172.21.15.69:0/3601315157 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588003d80 con 0x7f05a00ff2d0 2023-07-12T15:45:05.512 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.501+0000 7f05a5e0b700 1 -- 172.21.15.69:0/3601315157 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a01a1630 con 0x7f05a00ff2d0 2023-07-12T15:45:05.512 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.502+0000 7f05a5e0b700 1 -- 172.21.15.69:0/3601315157 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a0067150 con 0x7f05a00ff2d0 2023-07-12T15:45:05.512 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.503+0000 7f059cff9700 1 -- 172.21.15.69:0/3601315157 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f0588003ee0 con 0x7f05a00ff2d0 2023-07-12T15:45:05.513 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.503+0000 7f059cff9700 1 --2- 172.21.15.69:0/3601315157 >> v2:172.21.15.69:6800/1242716914 conn(0x7f058c06c810 0x7f058c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:05.513 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.503+0000 7f059cff9700 1 -- 172.21.15.69:0/3601315157 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(758..758 src has 1..758) v4 ==== 7671+0+0 (secure 0 0 0) 0x7f0588014070 con 0x7f05a00ff2d0 2023-07-12T15:45:05.513 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.503+0000 7f059effd700 1 --2- 172.21.15.69:0/3601315157 >> v2:172.21.15.69:6800/1242716914 conn(0x7f058c06c810 0x7f058c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:05.513 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.503+0000 7f059cff9700 1 -- 172.21.15.69:0/3601315157 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=759}) v3 -- 0x7f058c0778a0 con 0x7f05a00ff2d0 2023-07-12T15:45:05.513 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.503+0000 7f059effd700 1 --2- 172.21.15.69:0/3601315157 >> v2:172.21.15.69:6800/1242716914 conn(0x7f058c06c810 0x7f058c06ecc0 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 crypto rx=0x7f05a0195fc0 tx=0x7f0590009190 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:45:05.514 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.507+0000 7f059cff9700 1 -- 172.21.15.69:0/3601315157 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0588059f70 con 0x7f05a00ff2d0 2023-07-12T15:45:05.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:45:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:45:05.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:05 smithi145 ceph-mon[95801]: pgmap v1196: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:05.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2544431538' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]': finished 2023-07-12T15:45:05.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:05 smithi145 ceph-mon[95801]: osdmap e757: 8 total, 8 up, 8 in 2023-07-12T15:45:05.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:05 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2544431538' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:45:05.703 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:05.701+0000 7f05a5e0b700 1 -- 172.21.15.69:0/3601315157 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f05a0195d00 con 0x7f05a00ff2d0 2023-07-12T15:45:05.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:05 smithi069 ceph-mon[102777]: pgmap v1196: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:05.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2544431538' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]': finished 2023-07-12T15:45:05.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:05 smithi069 ceph-mon[102777]: osdmap e757: 8 total, 8 up, 8 in 2023-07-12T15:45:05.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:05 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2544431538' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:45:05.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:05 smithi069 ceph-mon[94241]: pgmap v1196: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:05.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2544431538' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]': finished 2023-07-12T15:45:05.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:05 smithi069 ceph-mon[94241]: osdmap e757: 8 total, 8 up, 8 in 2023-07-12T15:45:05.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:05 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2544431538' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:45:06.393 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.388+0000 7f059cff9700 1 -- 172.21.15.69:0/3601315157 <== mon.2 v2:172.21.15.69:3301/0 7 ==== osd_map(759..759 src has 1..759) v4 ==== 744+0+0 (secure 0 0 0) 0x7f0588016330 con 0x7f05a00ff2d0 2023-07-12T15:45:06.393 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.388+0000 7f059cff9700 1 -- 172.21.15.69:0/3601315157 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=760}) v3 -- 0x7f058c078390 con 0x7f05a00ff2d0 2023-07-12T15:45:06.393 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.390+0000 7f059cff9700 1 -- 172.21.15.69:0/3601315157 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v759) v1 ==== 221+0+0 (secure 0 0 0) 0x7f0588059b00 con 0x7f05a00ff2d0 2023-07-12T15:45:06.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:45:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:45:06] "GET /metrics HTTP/1.1" 200 226028 "" "Prometheus/2.43.0" 2023-07-12T15:45:06.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:06 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:06.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2544431538' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]': finished 2023-07-12T15:45:06.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:06 smithi069 ceph-mon[102777]: osdmap e758: 8 total, 8 up, 8 in 2023-07-12T15:45:06.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:06 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3601315157' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:45:06.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:06 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:45:06.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:06 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:06.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2544431538' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]': finished 2023-07-12T15:45:06.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:06 smithi069 ceph-mon[94241]: osdmap e758: 8 total, 8 up, 8 in 2023-07-12T15:45:06.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:06 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3601315157' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:45:06.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:06 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:45:06.524 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.522+0000 7f05a5e0b700 1 -- 172.21.15.69:0/3601315157 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f05a01a1850 con 0x7f05a00ff2d0 2023-07-12T15:45:06.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:06 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2544431538' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]': finished 2023-07-12T15:45:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:06 smithi145 ceph-mon[95801]: osdmap e758: 8 total, 8 up, 8 in 2023-07-12T15:45:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:06 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3601315157' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:45:06.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:06 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:45:06.965 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.962+0000 7f059cff9700 1 -- 172.21.15.69:0/3601315157 <== mon.2 v2:172.21.15.69:3301/0 9 ==== osd_map(760..760 src has 1..760) v4 ==== 744+0+0 (secure 0 0 0) 0x7f0588059b00 con 0x7f05a00ff2d0 2023-07-12T15:45:06.966 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.962+0000 7f059cff9700 1 -- 172.21.15.69:0/3601315157 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=761}) v3 -- 0x7f058c0787c0 con 0x7f05a00ff2d0 2023-07-12T15:45:06.970 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.968+0000 7f059cff9700 1 -- 172.21.15.69:0/3601315157 <== mon.2 v2:172.21.15.69:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v760) v1 ==== 221+0+0 (secure 0 0 0) 0x7f0588071030 con 0x7f05a00ff2d0 2023-07-12T15:45:06.974 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.971+0000 7f05a5e0b700 1 -- 172.21.15.69:0/3601315157 >> v2:172.21.15.69:6800/1242716914 conn(0x7f058c06c810 msgr2=0x7f058c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:06.974 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.971+0000 7f05a5e0b700 1 --2- 172.21.15.69:0/3601315157 >> v2:172.21.15.69:6800/1242716914 conn(0x7f058c06c810 0x7f058c06ecc0 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 crypto rx=0x7f05a0195fc0 tx=0x7f0590009190 comp rx=0 tx=0).stop 2023-07-12T15:45:06.974 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.971+0000 7f05a5e0b700 1 -- 172.21.15.69:0/3601315157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05a00ff2d0 msgr2=0x7f05a01949b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:06.974 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.971+0000 7f05a5e0b700 1 --2- 172.21.15.69:0/3601315157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05a00ff2d0 0x7f05a01949b0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7f058800eb30 tx=0x7f058800ee40 comp rx=0 tx=0).stop 2023-07-12T15:45:06.975 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.972+0000 7f05a5e0b700 1 -- 172.21.15.69:0/3601315157 shutdown_connections 2023-07-12T15:45:06.975 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.972+0000 7f05a5e0b700 1 --2- 172.21.15.69:0/3601315157 >> v2:172.21.15.69:6800/1242716914 conn(0x7f058c06c810 0x7f058c06ecc0 unknown :-1 s=CLOSED pgs=3231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:06.975 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.972+0000 7f05a5e0b700 1 --2- 172.21.15.69:0/3601315157 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05a0104930 0x7f05a0199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:06.975 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.972+0000 7f05a5e0b700 1 --2- 172.21.15.69:0/3601315157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05a00ff2d0 0x7f05a01949b0 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:06.975 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.972+0000 7f05a5e0b700 1 --2- 172.21.15.69:0/3601315157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05a00ffc80 0x7f05a0194ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:06.975 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.972+0000 7f05a5e0b700 1 -- 172.21.15.69:0/3601315157 >> 172.21.15.69:0/3601315157 conn(0x7f05a00748c0 msgr2=0x7f05a0105610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:45:06.975 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.972+0000 7f05a5e0b700 1 -- 172.21.15.69:0/3601315157 shutdown_connections 2023-07-12T15:45:06.976 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:06.972+0000 7f05a5e0b700 1 -- 172.21.15.69:0/3601315157 wait complete. 2023-07-12T15:45:06.976 INFO:tasks.workunit.client.0.smithi069.stderr:set-quota max_objects = 0 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 2023-07-12T15:45:06.985 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 204452 2023-07-12T15:45:06.985 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 0 -ne 0 ']' 2023-07-12T15:45:06.985 INFO:tasks.workunit.client.0.smithi069.stderr:+ true 2023-07-12T15:45:06.985 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put three /etc/passwd 2023-07-12T15:45:07.020 INFO:tasks.workunit.client.0.smithi069.stderr:++ uuidgen 2023-07-12T15:45:07.021 INFO:tasks.workunit.client.0.smithi069.stderr:+ pp=4d559536-7353-4f04-abe6-10edbbd13922 2023-07-12T15:45:07.021 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool create 4d559536-7353-4f04-abe6-10edbbd13922 12 2023-07-12T15:45:07.121 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.118+0000 7fc7bb792700 1 -- 172.21.15.69:0/567395880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7b40ffc30 msgr2=0x7fc7b4103bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:07.121 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.118+0000 7fc7bb792700 1 --2- 172.21.15.69:0/567395880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7b40ffc30 0x7fc7b4103bf0 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7fc7b00099c0 tx=0x7fc7b0009cd0 comp rx=0 tx=0).stop 2023-07-12T15:45:07.121 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.118+0000 7fc7bb792700 1 -- 172.21.15.69:0/567395880 shutdown_connections 2023-07-12T15:45:07.121 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.118+0000 7fc7bb792700 1 --2- 172.21.15.69:0/567395880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7b41042d0 0x7fc7b410bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:07.121 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.118+0000 7fc7bb792700 1 --2- 172.21.15.69:0/567395880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7b40ff290 0x7fc7b40ff660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:07.122 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.118+0000 7fc7bb792700 1 --2- 172.21.15.69:0/567395880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7b40ffc30 0x7fc7b4103bf0 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:07.122 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.118+0000 7fc7bb792700 1 -- 172.21.15.69:0/567395880 >> 172.21.15.69:0/567395880 conn(0x7fc7b40faae0 msgr2=0x7fc7b40fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:45:07.122 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.118+0000 7fc7bb792700 1 -- 172.21.15.69:0/567395880 shutdown_connections 2023-07-12T15:45:07.122 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.118+0000 7fc7bb792700 1 -- 172.21.15.69:0/567395880 wait complete. 2023-07-12T15:45:07.122 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7bb792700 1 Processor -- start 2023-07-12T15:45:07.122 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7bb792700 1 -- start start 2023-07-12T15:45:07.125 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7bb792700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7b40ff290 0x7fc7b419a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:07.125 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7bb792700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7b40ffc30 0x7fc7b419ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:07.125 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7bb792700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7b41042d0 0x7fc7b4194630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:07.125 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7bb792700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4194ca0 con 0x7fc7b40ff290 2023-07-12T15:45:07.125 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7bb792700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4194e10 con 0x7fc7b41042d0 2023-07-12T15:45:07.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7bb792700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4194fd0 con 0x7fc7b40ffc30 2023-07-12T15:45:07.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7b952e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7b40ff290 0x7fc7b419a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:07.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7b8d2d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7b40ffc30 0x7fc7b419ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:07.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7b952e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7b40ff290 0x7fc7b419a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47930/0 (socket says 172.21.15.69:47930) 2023-07-12T15:45:07.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7b8d2d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7b40ffc30 0x7fc7b419ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41360/0 (socket says 172.21.15.69:41360) 2023-07-12T15:45:07.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7b952e700 1 -- 172.21.15.69:0/2546706190 learned_addr learned my addr 172.21.15.69:0/2546706190 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:45:07.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7b9d2f700 1 --2- 172.21.15.69:0/2546706190 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7b41042d0 0x7fc7b4194630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:07.127 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7b8d2d700 1 -- 172.21.15.69:0/2546706190 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7b41042d0 msgr2=0x7fc7b4194630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:07.127 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7b8d2d700 1 --2- 172.21.15.69:0/2546706190 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7b41042d0 0x7fc7b4194630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:07.127 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7b8d2d700 1 -- 172.21.15.69:0/2546706190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7b40ff290 msgr2=0x7fc7b419a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:07.128 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7b8d2d700 1 --2- 172.21.15.69:0/2546706190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7b40ff290 0x7fc7b419a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:07.129 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7b8d2d700 1 -- 172.21.15.69:0/2546706190 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b00095f0 con 0x7fc7b40ffc30 2023-07-12T15:45:07.129 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7b952e700 1 --2- 172.21.15.69:0/2546706190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7b40ff290 0x7fc7b419a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:45:07.129 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7b8d2d700 1 --2- 172.21.15.69:0/2546706190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7b40ffc30 0x7fc7b419ab00 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7fc7b00098e0 tx=0x7fc7b00045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:45:07.129 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.119+0000 7fc7aa7fc700 1 -- 172.21.15.69:0/2546706190 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b001d070 con 0x7fc7b40ffc30 2023-07-12T15:45:07.129 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.120+0000 7fc7aa7fc700 1 -- 172.21.15.69:0/2546706190 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fc7b000bda0 con 0x7fc7b40ffc30 2023-07-12T15:45:07.129 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.120+0000 7fc7bb792700 1 -- 172.21.15.69:0/2546706190 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b4195250 con 0x7fc7b40ffc30 2023-07-12T15:45:07.129 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.120+0000 7fc7aa7fc700 1 -- 172.21.15.69:0/2546706190 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b000fa00 con 0x7fc7b40ffc30 2023-07-12T15:45:07.130 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.120+0000 7fc7bb792700 1 -- 172.21.15.69:0/2546706190 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b41a15b0 con 0x7fc7b40ffc30 2023-07-12T15:45:07.130 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.120+0000 7fc7bb792700 1 -- 172.21.15.69:0/2546706190 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b4067150 con 0x7fc7b40ffc30 2023-07-12T15:45:07.130 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.121+0000 7fc7aa7fc700 1 -- 172.21.15.69:0/2546706190 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7fc7b0003930 con 0x7fc7b40ffc30 2023-07-12T15:45:07.130 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.121+0000 7fc7aa7fc700 1 --2- 172.21.15.69:0/2546706190 >> v2:172.21.15.69:6800/1242716914 conn(0x7fc7a006c810 0x7fc7a006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:07.130 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.121+0000 7fc7aa7fc700 1 -- 172.21.15.69:0/2546706190 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(760..760 src has 1..760) v4 ==== 7671+0+0 (secure 0 0 0) 0x7fc7b008ce90 con 0x7fc7b40ffc30 2023-07-12T15:45:07.130 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.121+0000 7fc7b952e700 1 --2- 172.21.15.69:0/2546706190 >> v2:172.21.15.69:6800/1242716914 conn(0x7fc7a006c810 0x7fc7a006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:07.130 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.122+0000 7fc7b952e700 1 --2- 172.21.15.69:0/2546706190 >> v2:172.21.15.69:6800/1242716914 conn(0x7fc7a006c810 0x7fc7a006ecc0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4005950 tx=0x7fc7a400b500 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:45:07.131 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.126+0000 7fc7aa7fc700 1 -- 172.21.15.69:0/2546706190 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc7b005a760 con 0x7fc7b40ffc30 2023-07-12T15:45:07.316 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.314+0000 7fc7bb792700 1 -- 172.21.15.69:0/2546706190 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12} v 0) v1 -- 0x7fc7b4195b70 con 0x7fc7b40ffc30 2023-07-12T15:45:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:07 smithi145 ceph-mon[95801]: pgmap v1199: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:07 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]': finished 2023-07-12T15:45:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:07 smithi145 ceph-mon[95801]: osdmap e759: 8 total, 8 up, 8 in 2023-07-12T15:45:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:07 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3601315157' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:45:07.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:07 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:45:07.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:07 smithi145 ceph-mon[95801]: pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' no longer out of quota; removing NO_QUOTA flag 2023-07-12T15:45:07.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:07 smithi145 ceph-mon[95801]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2023-07-12T15:45:07.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:07 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]': finished 2023-07-12T15:45:07.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:07 smithi145 ceph-mon[95801]: osdmap e760: 8 total, 8 up, 8 in 2023-07-12T15:45:07.982 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:07.979+0000 7fc7aa7fc700 1 -- 172.21.15.69:0/2546706190 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]=0 pool '4d559536-7353-4f04-abe6-10edbbd13922' created v761) v1 ==== 176+0+0 (secure 0 0 0) 0x7fc7b005a2f0 con 0x7fc7b40ffc30 2023-07-12T15:45:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[94241]: pgmap v1199: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]': finished 2023-07-12T15:45:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[94241]: osdmap e759: 8 total, 8 up, 8 in 2023-07-12T15:45:07.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3601315157' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:45:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:45:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[94241]: pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' no longer out of quota; removing NO_QUOTA flag 2023-07-12T15:45:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[94241]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2023-07-12T15:45:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]': finished 2023-07-12T15:45:07.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[94241]: osdmap e760: 8 total, 8 up, 8 in 2023-07-12T15:45:07.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[102777]: pgmap v1199: 176 pgs: 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:07.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]': finished 2023-07-12T15:45:07.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[102777]: osdmap e759: 8 total, 8 up, 8 in 2023-07-12T15:45:07.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3601315157' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:45:07.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:45:07.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[102777]: pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' no longer out of quota; removing NO_QUOTA flag 2023-07-12T15:45:07.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[102777]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2023-07-12T15:45:07.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]': finished 2023-07-12T15:45:07.989 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:07 smithi069 ceph-mon[102777]: osdmap e760: 8 total, 8 up, 8 in 2023-07-12T15:45:08.111 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.108+0000 7fc7bb792700 1 -- 172.21.15.69:0/2546706190 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12} v 0) v1 -- 0x7fc7b41a1740 con 0x7fc7b40ffc30 2023-07-12T15:45:08.112 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.109+0000 7fc7aa7fc700 1 -- 172.21.15.69:0/2546706190 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]=0 pool '4d559536-7353-4f04-abe6-10edbbd13922' already exists v761) v1 ==== 183+0+0 (secure 0 0 0) 0x7fc7b00260c0 con 0x7fc7b40ffc30 2023-07-12T15:45:08.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.112+0000 7fc7bb792700 1 -- 172.21.15.69:0/2546706190 >> v2:172.21.15.69:6800/1242716914 conn(0x7fc7a006c810 msgr2=0x7fc7a006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:08.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.112+0000 7fc7bb792700 1 --2- 172.21.15.69:0/2546706190 >> v2:172.21.15.69:6800/1242716914 conn(0x7fc7a006c810 0x7fc7a006ecc0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4005950 tx=0x7fc7a400b500 comp rx=0 tx=0).stop 2023-07-12T15:45:08.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.112+0000 7fc7bb792700 1 -- 172.21.15.69:0/2546706190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7b40ffc30 msgr2=0x7fc7b419ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:08.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.112+0000 7fc7bb792700 1 --2- 172.21.15.69:0/2546706190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7b40ffc30 0x7fc7b419ab00 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7fc7b00098e0 tx=0x7fc7b00045e0 comp rx=0 tx=0).stop 2023-07-12T15:45:08.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.112+0000 7fc7bb792700 1 -- 172.21.15.69:0/2546706190 shutdown_connections 2023-07-12T15:45:08.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.112+0000 7fc7bb792700 1 --2- 172.21.15.69:0/2546706190 >> v2:172.21.15.69:6800/1242716914 conn(0x7fc7a006c810 0x7fc7a006ecc0 unknown :-1 s=CLOSED pgs=3233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:08.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.112+0000 7fc7bb792700 1 --2- 172.21.15.69:0/2546706190 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7b41042d0 0x7fc7b4194630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:08.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.112+0000 7fc7bb792700 1 --2- 172.21.15.69:0/2546706190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7b40ff290 0x7fc7b419a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:08.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.112+0000 7fc7bb792700 1 --2- 172.21.15.69:0/2546706190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7b40ffc30 0x7fc7b419ab00 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:08.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.112+0000 7fc7bb792700 1 -- 172.21.15.69:0/2546706190 >> 172.21.15.69:0/2546706190 conn(0x7fc7b40faae0 msgr2=0x7fc7b40fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:45:08.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.112+0000 7fc7bb792700 1 -- 172.21.15.69:0/2546706190 shutdown_connections 2023-07-12T15:45:08.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.112+0000 7fc7bb792700 1 -- 172.21.15.69:0/2546706190 wait complete. 2023-07-12T15:45:08.116 INFO:tasks.workunit.client.0.smithi069.stderr:pool '4d559536-7353-4f04-abe6-10edbbd13922' already exists 2023-07-12T15:45:08.124 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool application enable 4d559536-7353-4f04-abe6-10edbbd13922 rados 2023-07-12T15:45:08.228 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.226+0000 7fb69d3d3700 1 -- 172.21.15.69:0/1814000029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb698101210 msgr2=0x7fb6981015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:08.229 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.226+0000 7fb69d3d3700 1 --2- 172.21.15.69:0/1814000029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb698101210 0x7fb6981015e0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009990 tx=0x7fb68c009ca0 comp rx=0 tx=0).stop 2023-07-12T15:45:08.229 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.226+0000 7fb69d3d3700 1 -- 172.21.15.69:0/1814000029 shutdown_connections 2023-07-12T15:45:08.229 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.226+0000 7fb69d3d3700 1 --2- 172.21.15.69:0/1814000029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb69810a730 0x7fb69810cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:08.229 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.226+0000 7fb69d3d3700 1 --2- 172.21.15.69:0/1814000029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb698101210 0x7fb6981015e0 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:08.229 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.226+0000 7fb69d3d3700 1 --2- 172.21.15.69:0/1814000029 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb698101b20 0x7fb69810a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:08.230 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.226+0000 7fb69d3d3700 1 -- 172.21.15.69:0/1814000029 >> 172.21.15.69:0/1814000029 conn(0x7fb698075e10 msgr2=0x7fb698076210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:45:08.230 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.226+0000 7fb69d3d3700 1 -- 172.21.15.69:0/1814000029 shutdown_connections 2023-07-12T15:45:08.230 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.226+0000 7fb69d3d3700 1 -- 172.21.15.69:0/1814000029 wait complete. 2023-07-12T15:45:08.230 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.226+0000 7fb69d3d3700 1 Processor -- start 2023-07-12T15:45:08.232 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb69d3d3700 1 -- start start 2023-07-12T15:45:08.232 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb69d3d3700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb698101210 0x7fb69819b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:08.232 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb69d3d3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb698101b20 0x7fb69819bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:08.232 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb69d3d3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb69810a730 0x7fb698195740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:08.233 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb69d3d3700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb698195db0 con 0x7fb69810a730 2023-07-12T15:45:08.233 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb69d3d3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb698195f20 con 0x7fb698101b20 2023-07-12T15:45:08.233 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb69d3d3700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6981960e0 con 0x7fb698101210 2023-07-12T15:45:08.233 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb696ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb698101210 0x7fb69819b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:08.233 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb696ffd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb698101210 0x7fb69819b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41372/0 (socket says 172.21.15.69:41372) 2023-07-12T15:45:08.233 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb696ffd700 1 -- 172.21.15.69:0/3904751054 learned_addr learned my addr 172.21.15.69:0/3904751054 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:45:08.234 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb6977fe700 1 --2- 172.21.15.69:0/3904751054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb69810a730 0x7fb698195740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:08.234 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb6967fc700 1 --2- 172.21.15.69:0/3904751054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb698101b20 0x7fb69819bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:08.234 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb696ffd700 1 -- 172.21.15.69:0/3904751054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb698101b20 msgr2=0x7fb69819bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:08.234 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb696ffd700 1 --2- 172.21.15.69:0/3904751054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb698101b20 0x7fb69819bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:08.234 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb696ffd700 1 -- 172.21.15.69:0/3904751054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb69810a730 msgr2=0x7fb698195740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:08.234 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb696ffd700 1 --2- 172.21.15.69:0/3904751054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb69810a730 0x7fb698195740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:08.234 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb696ffd700 1 -- 172.21.15.69:0/3904751054 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c0095f0 con 0x7fb698101210 2023-07-12T15:45:08.234 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb6967fc700 1 --2- 172.21.15.69:0/3904751054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb698101b20 0x7fb69819bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-12T15:45:08.235 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.227+0000 7fb696ffd700 1 --2- 172.21.15.69:0/3904751054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb698101210 0x7fb69819b6d0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7fb68c006010 tx=0x7fb68c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:45:08.235 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.228+0000 7fb67ffff700 1 -- 172.21.15.69:0/3904751054 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c01d070 con 0x7fb698101210 2023-07-12T15:45:08.235 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.228+0000 7fb67ffff700 1 -- 172.21.15.69:0/3904751054 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fb68c00fbc0 con 0x7fb698101210 2023-07-12T15:45:08.235 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.228+0000 7fb69d3d3700 1 -- 172.21.15.69:0/3904751054 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb698196360 con 0x7fb698101210 2023-07-12T15:45:08.235 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.228+0000 7fb67ffff700 1 -- 172.21.15.69:0/3904751054 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c0173f0 con 0x7fb698101210 2023-07-12T15:45:08.235 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.228+0000 7fb69d3d3700 1 -- 172.21.15.69:0/3904751054 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6981a26c0 con 0x7fb698101210 2023-07-12T15:45:08.235 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.229+0000 7fb69d3d3700 1 -- 172.21.15.69:0/3904751054 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6980679e0 con 0x7fb698101210 2023-07-12T15:45:08.236 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.229+0000 7fb67ffff700 1 -- 172.21.15.69:0/3904751054 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7fb68c022470 con 0x7fb698101210 2023-07-12T15:45:08.236 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.229+0000 7fb67ffff700 1 --2- 172.21.15.69:0/3904751054 >> v2:172.21.15.69:6800/1242716914 conn(0x7fb68006c810 0x7fb68006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:08.236 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.230+0000 7fb67ffff700 1 -- 172.21.15.69:0/3904751054 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(761..761 src has 1..761) v4 ==== 8046+0+0 (secure 0 0 0) 0x7fb68c08d900 con 0x7fb698101210 2023-07-12T15:45:08.236 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.230+0000 7fb6967fc700 1 --2- 172.21.15.69:0/3904751054 >> v2:172.21.15.69:6800/1242716914 conn(0x7fb68006c810 0x7fb68006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:08.236 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.230+0000 7fb6967fc700 1 --2- 172.21.15.69:0/3904751054 >> v2:172.21.15.69:6800/1242716914 conn(0x7fb68006c810 0x7fb68006ecc0 secure :-1 s=READY pgs=3234 cs=0 l=1 rev1=1 crypto rx=0x7fb688005950 tx=0x7fb6880058e0 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:45:08.237 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.234+0000 7fb67ffff700 1 -- 172.21.15.69:0/3904751054 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb68c05b030 con 0x7fb698101210 2023-07-12T15:45:08.425 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.423+0000 7fb69d3d3700 1 -- 172.21.15.69:0/3904751054 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"} v 0) v1 -- 0x7fb698196c50 con 0x7fb698101210 2023-07-12T15:45:08.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2546706190' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]: dispatch 2023-07-12T15:45:08.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:08 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]: dispatch 2023-07-12T15:45:08.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:08 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]': finished 2023-07-12T15:45:08.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:08 smithi145 ceph-mon[95801]: osdmap e761: 8 total, 8 up, 8 in 2023-07-12T15:45:08.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2546706190' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]: dispatch 2023-07-12T15:45:08.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:08 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]: dispatch 2023-07-12T15:45:08.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:08 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3904751054' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]: dispatch 2023-07-12T15:45:08.852 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:08 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]: dispatch 2023-07-12T15:45:08.983 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:08.981+0000 7fb67ffff700 1 -- 172.21.15.69:0/3904751054 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]=0 enabled application 'rados' on pool '4d559536-7353-4f04-abe6-10edbbd13922' v762) v1 ==== 213+0+0 (secure 0 0 0) 0x7fb68c0270d0 con 0x7fb698101210 2023-07-12T15:45:08.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2546706190' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]: dispatch 2023-07-12T15:45:08.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]: dispatch 2023-07-12T15:45:08.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]': finished 2023-07-12T15:45:08.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[102777]: osdmap e761: 8 total, 8 up, 8 in 2023-07-12T15:45:08.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2546706190' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]: dispatch 2023-07-12T15:45:08.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]: dispatch 2023-07-12T15:45:08.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3904751054' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]: dispatch 2023-07-12T15:45:08.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]: dispatch 2023-07-12T15:45:08.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2546706190' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]: dispatch 2023-07-12T15:45:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]: dispatch 2023-07-12T15:45:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]': finished 2023-07-12T15:45:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[94241]: osdmap e761: 8 total, 8 up, 8 in 2023-07-12T15:45:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2546706190' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]: dispatch 2023-07-12T15:45:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pg_num": 12}]: dispatch 2023-07-12T15:45:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3904751054' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]: dispatch 2023-07-12T15:45:08.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:08 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]: dispatch 2023-07-12T15:45:09.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.114+0000 7fb69d3d3700 1 -- 172.21.15.69:0/3904751054 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"} v 0) v1 -- 0x7fb6981a2850 con 0x7fb698101210 2023-07-12T15:45:09.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:09 smithi145 ceph-mon[95801]: pgmap v1203: 188 pgs: 12 unknown, 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-12T15:45:09.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:09 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]': finished 2023-07-12T15:45:09.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:09 smithi145 ceph-mon[95801]: osdmap e762: 8 total, 8 up, 8 in 2023-07-12T15:45:09.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:09 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3904751054' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]: dispatch 2023-07-12T15:45:09.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:09 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]: dispatch 2023-07-12T15:45:09.985 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.983+0000 7fb67ffff700 1 -- 172.21.15.69:0/3904751054 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]=0 enabled application 'rados' on pool '4d559536-7353-4f04-abe6-10edbbd13922' v763) v1 ==== 213+0+0 (secure 0 0 0) 0x7fb68c05abc0 con 0x7fb698101210 2023-07-12T15:45:09.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:09 smithi069 ceph-mon[94241]: pgmap v1203: 188 pgs: 12 unknown, 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-12T15:45:09.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:09 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]': finished 2023-07-12T15:45:09.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:09 smithi069 ceph-mon[94241]: osdmap e762: 8 total, 8 up, 8 in 2023-07-12T15:45:09.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:09 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3904751054' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]: dispatch 2023-07-12T15:45:09.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:09 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]: dispatch 2023-07-12T15:45:09.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:09 smithi069 ceph-mon[102777]: pgmap v1203: 188 pgs: 12 unknown, 176 active+clean; 711 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-07-12T15:45:09.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:09 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]': finished 2023-07-12T15:45:09.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:09 smithi069 ceph-mon[102777]: osdmap e762: 8 total, 8 up, 8 in 2023-07-12T15:45:09.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:09 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3904751054' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]: dispatch 2023-07-12T15:45:09.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:09 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]: dispatch 2023-07-12T15:45:09.989 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.986+0000 7fb69d3d3700 1 -- 172.21.15.69:0/3904751054 >> v2:172.21.15.69:6800/1242716914 conn(0x7fb68006c810 msgr2=0x7fb68006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:09.989 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.986+0000 7fb69d3d3700 1 --2- 172.21.15.69:0/3904751054 >> v2:172.21.15.69:6800/1242716914 conn(0x7fb68006c810 0x7fb68006ecc0 secure :-1 s=READY pgs=3234 cs=0 l=1 rev1=1 crypto rx=0x7fb688005950 tx=0x7fb6880058e0 comp rx=0 tx=0).stop 2023-07-12T15:45:09.989 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.986+0000 7fb69d3d3700 1 -- 172.21.15.69:0/3904751054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb698101210 msgr2=0x7fb69819b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:09.989 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.986+0000 7fb69d3d3700 1 --2- 172.21.15.69:0/3904751054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb698101210 0x7fb69819b6d0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7fb68c006010 tx=0x7fb68c004730 comp rx=0 tx=0).stop 2023-07-12T15:45:09.989 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.986+0000 7fb69d3d3700 1 -- 172.21.15.69:0/3904751054 shutdown_connections 2023-07-12T15:45:09.990 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.986+0000 7fb69d3d3700 1 --2- 172.21.15.69:0/3904751054 >> v2:172.21.15.69:6800/1242716914 conn(0x7fb68006c810 0x7fb68006ecc0 unknown :-1 s=CLOSED pgs=3234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:09.990 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.986+0000 7fb69d3d3700 1 --2- 172.21.15.69:0/3904751054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb69810a730 0x7fb698195740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:09.990 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.986+0000 7fb69d3d3700 1 --2- 172.21.15.69:0/3904751054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb698101210 0x7fb69819b6d0 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:09.990 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.986+0000 7fb69d3d3700 1 --2- 172.21.15.69:0/3904751054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb698101b20 0x7fb69819bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:09.990 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.986+0000 7fb69d3d3700 1 -- 172.21.15.69:0/3904751054 >> 172.21.15.69:0/3904751054 conn(0x7fb698075e10 msgr2=0x7fb6980fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:45:09.990 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.986+0000 7fb69d3d3700 1 -- 172.21.15.69:0/3904751054 shutdown_connections 2023-07-12T15:45:09.990 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:09.986+0000 7fb69d3d3700 1 -- 172.21.15.69:0/3904751054 wait complete. 2023-07-12T15:45:09.991 INFO:tasks.workunit.client.0.smithi069.stderr:enabled application 'rados' on pool '4d559536-7353-4f04-abe6-10edbbd13922' 2023-07-12T15:45:10.001 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool set-quota 4d559536-7353-4f04-abe6-10edbbd13922 max_objects 10 2023-07-12T15:45:10.109 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.107+0000 7f7905614700 1 -- 172.21.15.69:0/2754382754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79000ff2b0 msgr2=0x7f79000ff680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:10.110 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.107+0000 7f7905614700 1 --2- 172.21.15.69:0/2754382754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79000ff2b0 0x7f79000ff680 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009920 tx=0x7f78f4009c30 comp rx=0 tx=0).stop 2023-07-12T15:45:10.110 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.107+0000 7f7905614700 1 -- 172.21.15.69:0/2754382754 shutdown_connections 2023-07-12T15:45:10.110 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.107+0000 7f7905614700 1 --2- 172.21.15.69:0/2754382754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79001042f0 0x7f790010bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:10.110 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.107+0000 7f7905614700 1 --2- 172.21.15.69:0/2754382754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79000ff2b0 0x7f79000ff680 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:10.110 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.107+0000 7f7905614700 1 --2- 172.21.15.69:0/2754382754 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79000ffc50 0x7f7900103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:10.111 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.107+0000 7f7905614700 1 -- 172.21.15.69:0/2754382754 >> 172.21.15.69:0/2754382754 conn(0x7f7900075e10 msgr2=0x7f7900076210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:45:10.111 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.107+0000 7f7905614700 1 -- 172.21.15.69:0/2754382754 shutdown_connections 2023-07-12T15:45:10.111 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.107+0000 7f7905614700 1 -- 172.21.15.69:0/2754382754 wait complete. 2023-07-12T15:45:10.111 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.107+0000 7f7905614700 1 Processor -- start 2023-07-12T15:45:10.113 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f7905614700 1 -- start start 2023-07-12T15:45:10.113 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f7905614700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79000ff2b0 0x7f790019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:10.113 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f7905614700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79000ffc50 0x7f790019be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:10.113 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f7905614700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79001042f0 0x7f79001958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:10.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f7905614700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7900196080 con 0x7f79000ffc50 2023-07-12T15:45:10.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f7905614700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79001961f0 con 0x7f79000ff2b0 2023-07-12T15:45:10.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f7905614700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79001963b0 con 0x7f79001042f0 2023-07-12T15:45:10.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f78fe7fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79000ffc50 0x7f790019be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:10.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f78ff7fe700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79001042f0 0x7f79001958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:10.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f78fe7fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79000ffc50 0x7f790019be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47960/0 (socket says 172.21.15.69:47960) 2023-07-12T15:45:10.114 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f78fe7fc700 1 -- 172.21.15.69:0/3663904404 learned_addr learned my addr 172.21.15.69:0/3663904404 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:45:10.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f78feffd700 1 --2- 172.21.15.69:0/3663904404 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79000ff2b0 0x7f790019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:10.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f78fe7fc700 1 -- 172.21.15.69:0/3663904404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79001042f0 msgr2=0x7f79001958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:10.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f78fe7fc700 1 --2- 172.21.15.69:0/3663904404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79001042f0 0x7f79001958b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:10.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f78fe7fc700 1 -- 172.21.15.69:0/3663904404 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79000ff2b0 msgr2=0x7f790019b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:10.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f78fe7fc700 1 --2- 172.21.15.69:0/3663904404 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79000ff2b0 0x7f790019b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:10.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f78fe7fc700 1 -- 172.21.15.69:0/3663904404 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f8009d00 con 0x7f79000ffc50 2023-07-12T15:45:10.115 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f78ff7fe700 1 --2- 172.21.15.69:0/3663904404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79001042f0 0x7f79001958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:45:10.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.108+0000 7f78fe7fc700 1 --2- 172.21.15.69:0/3663904404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79000ffc50 0x7f790019be10 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f78f000b4d0 tx=0x7f78f000b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:45:10.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.109+0000 7f78e7fff700 1 -- 172.21.15.69:0/3663904404 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f0011830 con 0x7f79000ffc50 2023-07-12T15:45:10.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.109+0000 7f78e7fff700 1 -- 172.21.15.69:0/3663904404 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f78f0011e20 con 0x7f79000ffc50 2023-07-12T15:45:10.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.109+0000 7f78e7fff700 1 -- 172.21.15.69:0/3663904404 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f0010ca0 con 0x7f79000ffc50 2023-07-12T15:45:10.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.109+0000 7f7905614700 1 -- 172.21.15.69:0/3663904404 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f4009520 con 0x7f79000ffc50 2023-07-12T15:45:10.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.109+0000 7f7905614700 1 -- 172.21.15.69:0/3663904404 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79001a2850 con 0x7f79000ffc50 2023-07-12T15:45:10.116 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.110+0000 7f7905614700 1 -- 172.21.15.69:0/3663904404 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79001085c0 con 0x7f79000ffc50 2023-07-12T15:45:10.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.110+0000 7f78e7fff700 1 -- 172.21.15.69:0/3663904404 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f78f0011990 con 0x7f79000ffc50 2023-07-12T15:45:10.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.110+0000 7f78e7fff700 1 --2- 172.21.15.69:0/3663904404 >> v2:172.21.15.69:6800/1242716914 conn(0x7f78e806c810 0x7f78e806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:45:10.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.110+0000 7f78e7fff700 1 -- 172.21.15.69:0/3663904404 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(763..763 src has 1..763) v4 ==== 8059+0+0 (secure 0 0 0) 0x7f78f008bb80 con 0x7f79000ffc50 2023-07-12T15:45:10.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.110+0000 7f78feffd700 1 --2- 172.21.15.69:0/3663904404 >> v2:172.21.15.69:6800/1242716914 conn(0x7f78e806c810 0x7f78e806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:45:10.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.111+0000 7f78feffd700 1 --2- 172.21.15.69:0/3663904404 >> v2:172.21.15.69:6800/1242716914 conn(0x7f78e806c810 0x7f78e806ecc0 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 crypto rx=0x7f78f400fac0 tx=0x7f78f4019040 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:45:10.117 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.115+0000 7f78e7fff700 1 -- 172.21.15.69:0/3663904404 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f78f0059350 con 0x7f79000ffc50 2023-07-12T15:45:10.306 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.304+0000 7f7905614700 1 -- 172.21.15.69:0/3663904404 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f7900196de0 con 0x7f79000ffc50 2023-07-12T15:45:10.990 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:10.988+0000 7f78e7fff700 1 -- 172.21.15.69:0/3663904404 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 4d559536-7353-4f04-abe6-10edbbd13922 v764) v1 ==== 223+0+0 (secure 0 0 0) 0x7f78f0058ee0 con 0x7f79000ffc50 2023-07-12T15:45:11.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.116+0000 7f7905614700 1 -- 172.21.15.69:0/3663904404 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f79000679e0 con 0x7f79000ffc50 2023-07-12T15:45:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:10 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]': finished 2023-07-12T15:45:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:10 smithi145 ceph-mon[95801]: osdmap e763: 8 total, 8 up, 8 in 2023-07-12T15:45:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:10 smithi145 ceph-mon[95801]: pgmap v1206: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 5.7 KiB/s wr, 1 op/s 2023-07-12T15:45:11.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:10 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3663904404' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:45:11.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:10 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]': finished 2023-07-12T15:45:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:10 smithi069 ceph-mon[102777]: osdmap e763: 8 total, 8 up, 8 in 2023-07-12T15:45:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:10 smithi069 ceph-mon[102777]: pgmap v1206: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 5.7 KiB/s wr, 1 op/s 2023-07-12T15:45:11.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:10 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3663904404' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:45:11.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:10 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "app": "rados"}]': finished 2023-07-12T15:45:11.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:10 smithi069 ceph-mon[94241]: osdmap e763: 8 total, 8 up, 8 in 2023-07-12T15:45:11.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:10 smithi069 ceph-mon[94241]: pgmap v1206: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 5.7 KiB/s wr, 1 op/s 2023-07-12T15:45:11.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:10 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3663904404' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:45:11.994 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.991+0000 7f78e7fff700 1 -- 172.21.15.69:0/3663904404 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 4d559536-7353-4f04-abe6-10edbbd13922 v765) v1 ==== 223+0+0 (secure 0 0 0) 0x7f78f0015030 con 0x7f79000ffc50 2023-07-12T15:45:11.997 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.995+0000 7f7905614700 1 -- 172.21.15.69:0/3663904404 >> v2:172.21.15.69:6800/1242716914 conn(0x7f78e806c810 msgr2=0x7f78e806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:11.997 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.995+0000 7f7905614700 1 --2- 172.21.15.69:0/3663904404 >> v2:172.21.15.69:6800/1242716914 conn(0x7f78e806c810 0x7f78e806ecc0 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 crypto rx=0x7f78f400fac0 tx=0x7f78f4019040 comp rx=0 tx=0).stop 2023-07-12T15:45:11.997 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.995+0000 7f7905614700 1 -- 172.21.15.69:0/3663904404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79000ffc50 msgr2=0x7f790019be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:45:11.998 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.995+0000 7f7905614700 1 --2- 172.21.15.69:0/3663904404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79000ffc50 0x7f790019be10 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f78f000b4d0 tx=0x7f78f000b7e0 comp rx=0 tx=0).stop 2023-07-12T15:45:11.998 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.995+0000 7f7905614700 1 -- 172.21.15.69:0/3663904404 shutdown_connections 2023-07-12T15:45:11.998 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.995+0000 7f7905614700 1 --2- 172.21.15.69:0/3663904404 >> v2:172.21.15.69:6800/1242716914 conn(0x7f78e806c810 0x7f78e806ecc0 unknown :-1 s=CLOSED pgs=3235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:11.998 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.995+0000 7f7905614700 1 --2- 172.21.15.69:0/3663904404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79001042f0 0x7f79001958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:11.998 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.995+0000 7f7905614700 1 --2- 172.21.15.69:0/3663904404 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79000ff2b0 0x7f790019b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:45:11.998 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.995+0000 7f7905614700 1 --2- 172.21.15.69:0/3663904404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79000ffc50 0x7f790019be10 secure :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f78f000b4d0 tx=0x7f78f000b7e0 comp rx=0 tx=0).stop 2023-07-12T15:45:11.998 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.995+0000 7f7905614700 1 -- 172.21.15.69:0/3663904404 >> 172.21.15.69:0/3663904404 conn(0x7f7900075e10 msgr2=0x7f7900102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:45:11.999 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.996+0000 7f7905614700 1 -- 172.21.15.69:0/3663904404 shutdown_connections 2023-07-12T15:45:11.999 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:45:11.996+0000 7f7905614700 1 -- 172.21.15.69:0/3663904404 wait complete. 2023-07-12T15:45:11.999 INFO:tasks.workunit.client.0.smithi069.stderr:set-quota max_objects = 10 for pool 4d559536-7353-4f04-abe6-10edbbd13922 2023-07-12T15:45:12.011 INFO:tasks.workunit.client.0.smithi069.stderr:+ sleep 30 2023-07-12T15:45:12.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3663904404' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]': finished 2023-07-12T15:45:12.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:11 smithi145 ceph-mon[95801]: osdmap e764: 8 total, 8 up, 8 in 2023-07-12T15:45:12.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:11 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3663904404' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:45:12.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3663904404' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]': finished 2023-07-12T15:45:12.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:11 smithi069 ceph-mon[102777]: osdmap e764: 8 total, 8 up, 8 in 2023-07-12T15:45:12.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:11 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3663904404' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:45:12.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3663904404' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]': finished 2023-07-12T15:45:12.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:11 smithi069 ceph-mon[94241]: osdmap e764: 8 total, 8 up, 8 in 2023-07-12T15:45:12.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:11 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3663904404' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]: dispatch 2023-07-12T15:45:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3663904404' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]': finished 2023-07-12T15:45:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:13 smithi145 ceph-mon[95801]: osdmap e765: 8 total, 8 up, 8 in 2023-07-12T15:45:13.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:13 smithi145 ceph-mon[95801]: pgmap v1209: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.5 KiB/s wr, 1 op/s 2023-07-12T15:45:13.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3663904404' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]': finished 2023-07-12T15:45:13.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:13 smithi069 ceph-mon[102777]: osdmap e765: 8 total, 8 up, 8 in 2023-07-12T15:45:13.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:13 smithi069 ceph-mon[102777]: pgmap v1209: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.5 KiB/s wr, 1 op/s 2023-07-12T15:45:13.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3663904404' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "field": "max_objects", "val": "10"}]': finished 2023-07-12T15:45:13.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:13 smithi069 ceph-mon[94241]: osdmap e765: 8 total, 8 up, 8 in 2023-07-12T15:45:13.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:13 smithi069 ceph-mon[94241]: pgmap v1209: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.5 KiB/s wr, 1 op/s 2023-07-12T15:45:14.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:14 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:45:14.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:14 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:45:14.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:14 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:45:14.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:14 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:45:14.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:14 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:45:14.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:14 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:45:15.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:45:15 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:45:15.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:15 smithi145 ceph-mon[95801]: pgmap v1210: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 4.2 KiB/s wr, 1 op/s 2023-07-12T15:45:15.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:15 smithi069 ceph-mon[102777]: pgmap v1210: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 4.2 KiB/s wr, 1 op/s 2023-07-12T15:45:15.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:15 smithi069 ceph-mon[94241]: pgmap v1210: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 4.2 KiB/s wr, 1 op/s 2023-07-12T15:45:16.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:45:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:45:16] "GET /metrics HTTP/1.1" 200 228459 "" "Prometheus/2.43.0" 2023-07-12T15:45:16.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:16 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:16.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:16 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:16.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:16 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:17.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:17 smithi145 ceph-mon[95801]: pgmap v1211: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 815 B/s rd, 0 op/s 2023-07-12T15:45:17.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:17 smithi069 ceph-mon[102777]: pgmap v1211: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 815 B/s rd, 0 op/s 2023-07-12T15:45:17.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:17 smithi069 ceph-mon[94241]: pgmap v1211: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 815 B/s rd, 0 op/s 2023-07-12T15:45:19.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:19 smithi145 ceph-mon[95801]: pgmap v1212: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:45:19.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:19 smithi069 ceph-mon[102777]: pgmap v1212: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:45:19.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:19 smithi069 ceph-mon[94241]: pgmap v1212: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:45:21.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:21 smithi145 ceph-mon[95801]: pgmap v1213: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-07-12T15:45:21.985 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:21 smithi069 ceph-mon[102777]: pgmap v1213: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-07-12T15:45:21.986 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:21 smithi069 ceph-mon[94241]: pgmap v1213: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-07-12T15:45:23.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:22 smithi069 ceph-mon[102777]: pgmap v1214: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 996 B/s rd, 0 op/s 2023-07-12T15:45:23.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:22 smithi069 ceph-mon[94241]: pgmap v1214: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 996 B/s rd, 0 op/s 2023-07-12T15:45:23.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:22 smithi145 ceph-mon[95801]: pgmap v1214: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 996 B/s rd, 0 op/s 2023-07-12T15:45:25.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:45:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:45:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:25 smithi145 ceph-mon[95801]: pgmap v1215: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:25.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:25 smithi069 ceph-mon[102777]: pgmap v1215: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:25.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:25 smithi069 ceph-mon[94241]: pgmap v1215: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:26.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:26 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:26 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:26.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:45:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:45:26] "GET /metrics HTTP/1.1" 200 228459 "" "Prometheus/2.43.0" 2023-07-12T15:45:26.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:26 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:27.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:27 smithi145 ceph-mon[95801]: pgmap v1216: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:27.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:27 smithi069 ceph-mon[102777]: pgmap v1216: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:27.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:27 smithi069 ceph-mon[94241]: pgmap v1216: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:29 smithi145 ceph-mon[95801]: pgmap v1217: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:45:29.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:45:29.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:29 smithi069 ceph-mon[102777]: pgmap v1217: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:45:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:45:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:29 smithi069 ceph-mon[94241]: pgmap v1217: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:45:29.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:45:31.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:31 smithi145 ceph-mon[95801]: pgmap v1218: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:31.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:31 smithi069 ceph-mon[102777]: pgmap v1218: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:31.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:31 smithi069 ceph-mon[94241]: pgmap v1218: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:33.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:32 smithi069 ceph-mon[102777]: pgmap v1219: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:45:33.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:32 smithi069 ceph-mon[94241]: pgmap v1219: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:45:33.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:32 smithi145 ceph-mon[95801]: pgmap v1219: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:45:34.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:33 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:45:34.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:33 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:45:34.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:33 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:45:34.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:33 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:45:34.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:33 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:45:34.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:33 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:45:34.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:33 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:45:34.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:33 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:45:34.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:33 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:45:34.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:33 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:45:34.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:33 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:45:34.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:33 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:45:35.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:34 smithi069 ceph-mon[102777]: pgmap v1220: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:35.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:34 smithi069 ceph-mon[94241]: pgmap v1220: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:35.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:34 smithi145 ceph-mon[95801]: pgmap v1220: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:35.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:45:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:45:36.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:35 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:36.236 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:45:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:45:36] "GET /metrics HTTP/1.1" 200 228460 "" "Prometheus/2.43.0" 2023-07-12T15:45:36.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:35 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:36.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:35 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:37.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:36 smithi145 ceph-mon[95801]: pgmap v1221: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:37.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:36 smithi069 ceph-mon[94241]: pgmap v1221: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:37.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:36 smithi069 ceph-mon[102777]: pgmap v1221: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:39.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:39 smithi145 ceph-mon[95801]: pgmap v1222: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:45:39.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:39 smithi069 ceph-mon[94241]: pgmap v1222: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:45:39.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:39 smithi069 ceph-mon[102777]: pgmap v1222: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:45:41.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:41 smithi145 ceph-mon[95801]: pgmap v1223: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:41.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:41 smithi069 ceph-mon[102777]: pgmap v1223: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:41.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:41 smithi069 ceph-mon[94241]: pgmap v1223: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:45:42.012 INFO:tasks.workunit.client.0.smithi069.stderr:++ seq 1 10 2023-07-12T15:45:42.013 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:45:42.013 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p 4d559536-7353-4f04-abe6-10edbbd13922 put obj1 /etc/passwd 2023-07-12T15:45:42.054 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:45:42.054 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p 4d559536-7353-4f04-abe6-10edbbd13922 put obj2 /etc/passwd 2023-07-12T15:45:42.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:45:42.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p 4d559536-7353-4f04-abe6-10edbbd13922 put obj3 /etc/passwd 2023-07-12T15:45:42.127 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:45:42.128 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p 4d559536-7353-4f04-abe6-10edbbd13922 put obj4 /etc/passwd 2023-07-12T15:45:42.162 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:45:42.162 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p 4d559536-7353-4f04-abe6-10edbbd13922 put obj5 /etc/passwd 2023-07-12T15:45:42.198 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:45:42.198 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p 4d559536-7353-4f04-abe6-10edbbd13922 put obj6 /etc/passwd 2023-07-12T15:45:42.232 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:45:42.232 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p 4d559536-7353-4f04-abe6-10edbbd13922 put obj7 /etc/passwd 2023-07-12T15:45:42.266 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:45:42.267 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p 4d559536-7353-4f04-abe6-10edbbd13922 put obj8 /etc/passwd 2023-07-12T15:45:42.301 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:45:42.301 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p 4d559536-7353-4f04-abe6-10edbbd13922 put obj9 /etc/passwd 2023-07-12T15:45:42.336 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in `seq 1 10` 2023-07-12T15:45:42.336 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p 4d559536-7353-4f04-abe6-10edbbd13922 put obj10 /etc/passwd 2023-07-12T15:45:42.371 INFO:tasks.workunit.client.0.smithi069.stderr:+ sleep 30 2023-07-12T15:45:43.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:42 smithi069 ceph-mon[102777]: pgmap v1224: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:43.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:42 smithi069 ceph-mon[94241]: pgmap v1224: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:43.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:42 smithi145 ceph-mon[95801]: pgmap v1224: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:44.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:43 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:45:44.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:43 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:45:44.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:43 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:45:45.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:44 smithi069 ceph-mon[102777]: pgmap v1225: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:45.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:44 smithi069 ceph-mon[94241]: pgmap v1225: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:45.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:44 smithi145 ceph-mon[95801]: pgmap v1225: 188 pgs: 188 active+clean; 732 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:45:45.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:45:45 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:45:46.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:45 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:46.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:45 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:45 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:46.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:45:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:45:46] "GET /metrics HTTP/1.1" 200 228437 "" "Prometheus/2.43.0" 2023-07-12T15:45:47.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:46 smithi145 ceph-mon[95801]: pgmap v1226: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s 2023-07-12T15:45:47.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:46 smithi069 ceph-mon[102777]: pgmap v1226: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s 2023-07-12T15:45:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:46 smithi069 ceph-mon[94241]: pgmap v1226: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s 2023-07-12T15:45:49.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:49 smithi145 ceph-mon[95801]: pgmap v1227: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 9.2 KiB/s wr, 1 op/s 2023-07-12T15:45:49.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:49 smithi069 ceph-mon[102777]: pgmap v1227: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 9.2 KiB/s wr, 1 op/s 2023-07-12T15:45:49.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:49 smithi069 ceph-mon[94241]: pgmap v1227: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 9.2 KiB/s wr, 1 op/s 2023-07-12T15:45:51.606 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:51 smithi145 ceph-mon[95801]: pgmap v1228: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s 2023-07-12T15:45:51.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:51 smithi069 ceph-mon[102777]: pgmap v1228: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s 2023-07-12T15:45:51.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:51 smithi069 ceph-mon[94241]: pgmap v1228: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.2 KiB/s wr, 2 op/s 2023-07-12T15:45:52.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:52 smithi145 ceph-mon[95801]: pool '4d559536-7353-4f04-abe6-10edbbd13922' is full (reached quota's max_objects: 10) 2023-07-12T15:45:52.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:52 smithi145 ceph-mon[95801]: Health check failed: 1 pool(s) full (POOL_FULL) 2023-07-12T15:45:52.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:52 smithi145 ceph-mon[95801]: osdmap e766: 8 total, 8 up, 8 in 2023-07-12T15:45:52.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:52 smithi069 ceph-mon[102777]: pool '4d559536-7353-4f04-abe6-10edbbd13922' is full (reached quota's max_objects: 10) 2023-07-12T15:45:52.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:52 smithi069 ceph-mon[102777]: Health check failed: 1 pool(s) full (POOL_FULL) 2023-07-12T15:45:52.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:52 smithi069 ceph-mon[102777]: osdmap e766: 8 total, 8 up, 8 in 2023-07-12T15:45:52.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:52 smithi069 ceph-mon[94241]: pool '4d559536-7353-4f04-abe6-10edbbd13922' is full (reached quota's max_objects: 10) 2023-07-12T15:45:52.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:52 smithi069 ceph-mon[94241]: Health check failed: 1 pool(s) full (POOL_FULL) 2023-07-12T15:45:52.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:52 smithi069 ceph-mon[94241]: osdmap e766: 8 total, 8 up, 8 in 2023-07-12T15:45:53.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:53 smithi145 ceph-mon[95801]: pgmap v1230: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2023-07-12T15:45:53.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:53 smithi069 ceph-mon[94241]: pgmap v1230: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2023-07-12T15:45:53.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:53 smithi069 ceph-mon[102777]: pgmap v1230: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2023-07-12T15:45:55.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:45:55 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:45:55.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:55 smithi145 ceph-mon[95801]: pgmap v1231: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2023-07-12T15:45:55.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:55 smithi069 ceph-mon[94241]: pgmap v1231: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2023-07-12T15:45:55.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:55 smithi069 ceph-mon[102777]: pgmap v1231: 188 pgs: 188 active+clean; 840 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2023-07-12T15:45:56.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:56 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:56.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:56 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:56.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:45:56 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:45:56] "GET /metrics HTTP/1.1" 200 228437 "" "Prometheus/2.43.0" 2023-07-12T15:45:56.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:56 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:45:57.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:57 smithi069 ceph-mon[102777]: pgmap v1232: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:45:57.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:57 smithi069 ceph-mon[94241]: pgmap v1232: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:45:57.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:57 smithi145 ceph-mon[95801]: pgmap v1232: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:45:59.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:59 smithi069 ceph-mon[102777]: pgmap v1233: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:45:59.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:59 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:45:59.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:45:59 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:45:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:59 smithi069 ceph-mon[94241]: pgmap v1233: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:45:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:59 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:45:59.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:45:59 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:45:59.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:59 smithi145 ceph-mon[95801]: pgmap v1233: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:45:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:59 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:45:59.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:45:59 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:46:01.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:01 smithi069 ceph-mon[102777]: pgmap v1234: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:46:01.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:01 smithi069 ceph-mon[94241]: pgmap v1234: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:46:01.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:01 smithi145 ceph-mon[95801]: pgmap v1234: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-07-12T15:46:03.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:02 smithi069 ceph-mon[102777]: pgmap v1235: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-12T15:46:03.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:02 smithi069 ceph-mon[94241]: pgmap v1235: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-12T15:46:03.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:02 smithi145 ceph-mon[95801]: pgmap v1235: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2023-07-12T15:46:05.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:46:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:46:05.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:05 smithi145 ceph-mon[95801]: pgmap v1236: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:05.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:05 smithi069 ceph-mon[102777]: pgmap v1236: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:05.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:05 smithi069 ceph-mon[94241]: pgmap v1236: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:06.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:06 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:06.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:06 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:06.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:06 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:46:06] "GET /metrics HTTP/1.1" 200 228503 "" "Prometheus/2.43.0" 2023-07-12T15:46:06.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:06 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:07.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:07 smithi145 ceph-mon[95801]: pgmap v1237: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:07.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:07 smithi069 ceph-mon[102777]: pgmap v1237: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:07.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:07 smithi069 ceph-mon[94241]: pgmap v1237: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:09.602 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:09 smithi145 ceph-mon[95801]: pgmap v1238: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:46:09.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:09 smithi069 ceph-mon[102777]: pgmap v1238: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:46:09.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:09 smithi069 ceph-mon[94241]: pgmap v1238: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:46:11.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:11 smithi069 ceph-mon[102777]: pgmap v1239: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:11.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:11 smithi069 ceph-mon[94241]: pgmap v1239: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:11.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:11 smithi145 ceph-mon[95801]: pgmap v1239: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:12.372 INFO:tasks.workunit.client.0.smithi069.stderr:+ rados -p e6659867-2dff-445e-9a1e-87d1f41b1a67 put threemore /etc/passwd 2023-07-12T15:46:12.412 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool set-quota e6659867-2dff-445e-9a1e-87d1f41b1a67 max_bytes 0 2023-07-12T15:46:12.519 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.517+0000 7f81c630f700 1 -- 172.21.15.69:0/383503499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81c00ff280 msgr2=0x7f81c00ff650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:12.520 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.517+0000 7f81c630f700 1 --2- 172.21.15.69:0/383503499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81c00ff280 0x7f81c00ff650 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f81b40099c0 tx=0x7f81b4009cd0 comp rx=0 tx=0).stop 2023-07-12T15:46:12.520 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.517+0000 7f81c630f700 1 -- 172.21.15.69:0/383503499 shutdown_connections 2023-07-12T15:46:12.520 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.517+0000 7f81c630f700 1 --2- 172.21.15.69:0/383503499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81c01042c0 0x7f81c010be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:12.520 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.517+0000 7f81c630f700 1 --2- 172.21.15.69:0/383503499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81c00ff280 0x7f81c00ff650 unknown :-1 s=CLOSED pgs=2776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:12.521 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.517+0000 7f81c630f700 1 --2- 172.21.15.69:0/383503499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c00ffc20 0x7f81c0103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:12.521 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.517+0000 7f81c630f700 1 -- 172.21.15.69:0/383503499 >> 172.21.15.69:0/383503499 conn(0x7f81c00faa90 msgr2=0x7f81c00fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:46:12.521 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.517+0000 7f81c630f700 1 -- 172.21.15.69:0/383503499 shutdown_connections 2023-07-12T15:46:12.521 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.517+0000 7f81c630f700 1 -- 172.21.15.69:0/383503499 wait complete. 2023-07-12T15:46:12.522 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.517+0000 7f81c630f700 1 Processor -- start 2023-07-12T15:46:12.522 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c630f700 1 -- start start 2023-07-12T15:46:12.524 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c630f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81c00ff280 0x7f81c0100c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:12.524 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c630f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c00ffc20 0x7f81c01011c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:12.524 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c630f700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81c01042c0 0x7f81c01a3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:12.524 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c630f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c01019f0 con 0x7f81c00ffc20 2023-07-12T15:46:12.524 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c630f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c01a4090 con 0x7f81c00ff280 2023-07-12T15:46:12.525 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c630f700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c01a41d0 con 0x7f81c01042c0 2023-07-12T15:46:12.525 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81bf7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c00ffc20 0x7f81c01011c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:12.525 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c48ac700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81c01042c0 0x7f81c01a3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:12.525 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81bf7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c00ffc20 0x7f81c01011c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48216/0 (socket says 172.21.15.69:48216) 2023-07-12T15:46:12.525 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81bf7fe700 1 -- 172.21.15.69:0/2366416202 learned_addr learned my addr 172.21.15.69:0/2366416202 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:46:12.525 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81bffff700 1 --2- 172.21.15.69:0/2366416202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81c00ff280 0x7f81c0100c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:12.525 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c48ac700 1 -- 172.21.15.69:0/2366416202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81c00ff280 msgr2=0x7f81c0100c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:12.526 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c48ac700 1 --2- 172.21.15.69:0/2366416202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81c00ff280 0x7f81c0100c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:12.526 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c48ac700 1 -- 172.21.15.69:0/2366416202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c00ffc20 msgr2=0x7f81c01011c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:12.526 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c48ac700 1 --2- 172.21.15.69:0/2366416202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c00ffc20 0x7f81c01011c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:12.526 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c48ac700 1 -- 172.21.15.69:0/2366416202 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b0009520 con 0x7f81c01042c0 2023-07-12T15:46:12.526 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81bf7fe700 1 --2- 172.21.15.69:0/2366416202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c00ffc20 0x7f81c01011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:46:12.526 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.518+0000 7f81c48ac700 1 --2- 172.21.15.69:0/2366416202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81c01042c0 0x7f81c01a3b50 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f81b8007f10 tx=0x7f81b800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:46:12.527 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.519+0000 7f81bd7fa700 1 -- 172.21.15.69:0/2366416202 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8009ea0 con 0x7f81c01042c0 2023-07-12T15:46:12.527 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.519+0000 7f81bd7fa700 1 -- 172.21.15.69:0/2366416202 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f81b800f040 con 0x7f81c01042c0 2023-07-12T15:46:12.527 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.519+0000 7f81c630f700 1 -- 172.21.15.69:0/2366416202 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b40095f0 con 0x7f81c01042c0 2023-07-12T15:46:12.527 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.519+0000 7f81bd7fa700 1 -- 172.21.15.69:0/2366416202 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8004360 con 0x7f81c01042c0 2023-07-12T15:46:12.527 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.519+0000 7f81c630f700 1 -- 172.21.15.69:0/2366416202 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c01a4780 con 0x7f81c01042c0 2023-07-12T15:46:12.527 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.520+0000 7f81c630f700 1 -- 172.21.15.69:0/2366416202 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c0067e30 con 0x7f81c01042c0 2023-07-12T15:46:12.527 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.520+0000 7f81bd7fa700 1 -- 172.21.15.69:0/2366416202 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f81b8003a00 con 0x7f81c01042c0 2023-07-12T15:46:12.528 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.520+0000 7f81bd7fa700 1 --2- 172.21.15.69:0/2366416202 >> v2:172.21.15.69:6800/1242716914 conn(0x7f81a806c810 0x7f81a806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:12.528 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.521+0000 7f81bd7fa700 1 -- 172.21.15.69:0/2366416202 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(766..766 src has 1..766) v4 ==== 8059+0+0 (secure 0 0 0) 0x7f81b808baf0 con 0x7f81c01042c0 2023-07-12T15:46:12.528 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.521+0000 7f81bffff700 1 --2- 172.21.15.69:0/2366416202 >> v2:172.21.15.69:6800/1242716914 conn(0x7f81a806c810 0x7f81a806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:12.529 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.521+0000 7f81bd7fa700 1 -- 172.21.15.69:0/2366416202 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=767}) v3 -- 0x7f81a80780c0 con 0x7f81c01042c0 2023-07-12T15:46:12.529 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.521+0000 7f81bffff700 1 --2- 172.21.15.69:0/2366416202 >> v2:172.21.15.69:6800/1242716914 conn(0x7f81a806c810 0x7f81a806ecc0 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7f81b4006010 tx=0x7f81b400b540 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:46:12.530 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.526+0000 7f81bd7fa700 1 -- 172.21.15.69:0/2366416202 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f81b8019020 con 0x7f81c01042c0 2023-07-12T15:46:12.715 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.713+0000 7f81c630f700 1 -- 172.21.15.69:0/2366416202 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f81c01a49a0 con 0x7f81c01042c0 2023-07-12T15:46:12.988 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.986+0000 7f81bd7fa700 1 -- 172.21.15.69:0/2366416202 <== mon.2 v2:172.21.15.69:3301/0 7 ==== osd_map(767..767 src has 1..767) v4 ==== 744+0+0 (secure 0 0 0) 0x7f81b8004ad0 con 0x7f81c01042c0 2023-07-12T15:46:12.989 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.986+0000 7f81bd7fa700 1 -- 172.21.15.69:0/2366416202 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=768}) v3 -- 0x7f81a8078c50 con 0x7f81c01042c0 2023-07-12T15:46:12.993 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:12.990+0000 7f81bd7fa700 1 -- 172.21.15.69:0/2366416202 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v767) v1 ==== 217+0+0 (secure 0 0 0) 0x7f81b8019020 con 0x7f81c01042c0 2023-07-12T15:46:13.118 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:13.115+0000 7f81c630f700 1 -- 172.21.15.69:0/2366416202 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f81c01a4bb0 con 0x7f81c01042c0 2023-07-12T15:46:13.235 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:12 smithi069 ceph-mon[102777]: pgmap v1240: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:13.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:12 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2366416202' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:46:13.236 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:12 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:46:13.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:12 smithi069 ceph-mon[94241]: pgmap v1240: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:13.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:12 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2366416202' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:46:13.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:12 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:46:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:12 smithi145 ceph-mon[95801]: pgmap v1240: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:12 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2366416202' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:46:13.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:12 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:46:13.997 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:13.993+0000 7f81bd7fa700 1 -- 172.21.15.69:0/2366416202 <== mon.2 v2:172.21.15.69:3301/0 9 ==== osd_map(768..768 src has 1..768) v4 ==== 744+0+0 (secure 0 0 0) 0x7f81b8059470 con 0x7f81c01042c0 2023-07-12T15:46:13.998 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:13.993+0000 7f81bd7fa700 1 -- 172.21.15.69:0/2366416202 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=769}) v3 -- 0x7f81a8079210 con 0x7f81c01042c0 2023-07-12T15:46:14.000 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:13.998+0000 7f81bd7fa700 1 -- 172.21.15.69:0/2366416202 <== mon.2 v2:172.21.15.69:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v768) v1 ==== 217+0+0 (secure 0 0 0) 0x7f81b8019030 con 0x7f81c01042c0 2023-07-12T15:46:14.003 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.001+0000 7f81c630f700 1 -- 172.21.15.69:0/2366416202 >> v2:172.21.15.69:6800/1242716914 conn(0x7f81a806c810 msgr2=0x7f81a806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:14.004 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.001+0000 7f81c630f700 1 --2- 172.21.15.69:0/2366416202 >> v2:172.21.15.69:6800/1242716914 conn(0x7f81a806c810 0x7f81a806ecc0 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7f81b4006010 tx=0x7f81b400b540 comp rx=0 tx=0).stop 2023-07-12T15:46:14.004 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.001+0000 7f81c630f700 1 -- 172.21.15.69:0/2366416202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81c01042c0 msgr2=0x7f81c01a3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:14.004 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.001+0000 7f81c630f700 1 --2- 172.21.15.69:0/2366416202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81c01042c0 0x7f81c01a3b50 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f81b8007f10 tx=0x7f81b800d3b0 comp rx=0 tx=0).stop 2023-07-12T15:46:14.004 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.001+0000 7f81c630f700 1 -- 172.21.15.69:0/2366416202 shutdown_connections 2023-07-12T15:46:14.004 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.001+0000 7f81c630f700 1 --2- 172.21.15.69:0/2366416202 >> v2:172.21.15.69:6800/1242716914 conn(0x7f81a806c810 0x7f81a806ecc0 unknown :-1 s=CLOSED pgs=3247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:14.004 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.001+0000 7f81c630f700 1 --2- 172.21.15.69:0/2366416202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81c01042c0 0x7f81c01a3b50 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:14.005 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.001+0000 7f81c630f700 1 --2- 172.21.15.69:0/2366416202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81c00ff280 0x7f81c0100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:14.005 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.001+0000 7f81c630f700 1 --2- 172.21.15.69:0/2366416202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c00ffc20 0x7f81c01011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:14.005 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.001+0000 7f81c630f700 1 -- 172.21.15.69:0/2366416202 >> 172.21.15.69:0/2366416202 conn(0x7f81c00faa90 msgr2=0x7f81c0069110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:46:14.005 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.001+0000 7f81c630f700 1 -- 172.21.15.69:0/2366416202 shutdown_connections 2023-07-12T15:46:14.005 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.001+0000 7f81c630f700 1 -- 172.21.15.69:0/2366416202 wait complete. 2023-07-12T15:46:14.005 INFO:tasks.workunit.client.0.smithi069.stderr:set-quota max_bytes = 0 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 2023-07-12T15:46:14.015 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool set-quota e6659867-2dff-445e-9a1e-87d1f41b1a67 max_objects 0 2023-07-12T15:46:14.122 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.118+0000 7fd0da9d0700 1 -- 172.21.15.69:0/4011555644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d4104930 msgr2=0x7fd0d4104d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:14.122 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.118+0000 7fd0da9d0700 1 --2- 172.21.15.69:0/4011555644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d4104930 0x7fd0d4104d00 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7fd0c40098f0 tx=0x7fd0c4009c00 comp rx=0 tx=0).stop 2023-07-12T15:46:14.122 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.118+0000 7fd0da9d0700 1 -- 172.21.15.69:0/4011555644 shutdown_connections 2023-07-12T15:46:14.122 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.118+0000 7fd0da9d0700 1 --2- 172.21.15.69:0/4011555644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40ffc80 0x7fd0d410b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:14.123 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.118+0000 7fd0da9d0700 1 --2- 172.21.15.69:0/4011555644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d4104930 0x7fd0d4104d00 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:14.123 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.118+0000 7fd0da9d0700 1 --2- 172.21.15.69:0/4011555644 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0d40ff2d0 0x7fd0d40ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:14.123 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.118+0000 7fd0da9d0700 1 -- 172.21.15.69:0/4011555644 >> 172.21.15.69:0/4011555644 conn(0x7fd0d40748c0 msgr2=0x7fd0d4074cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:46:14.123 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.118+0000 7fd0da9d0700 1 -- 172.21.15.69:0/4011555644 shutdown_connections 2023-07-12T15:46:14.123 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.118+0000 7fd0da9d0700 1 -- 172.21.15.69:0/4011555644 wait complete. 2023-07-12T15:46:14.123 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.118+0000 7fd0da9d0700 1 Processor -- start 2023-07-12T15:46:14.123 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.118+0000 7fd0da9d0700 1 -- start start 2023-07-12T15:46:14.124 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.118+0000 7fd0da9d0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40ff2d0 0x7fd0d419b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:14.124 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.118+0000 7fd0da9d0700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d40ffc80 0x7fd0d419bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:14.124 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0da9d0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0d4104930 0x7fd0d4195750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:14.124 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0da9d0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d4195dc0 con 0x7fd0d40ff2d0 2023-07-12T15:46:14.124 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0da9d0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d4195f30 con 0x7fd0d4104930 2023-07-12T15:46:14.124 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0da9d0700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d41960f0 con 0x7fd0d40ffc80 2023-07-12T15:46:14.124 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0d3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40ff2d0 0x7fd0d419b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:14.124 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0cbfff700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d40ffc80 0x7fd0d419bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:14.125 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0d3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40ff2d0 0x7fd0d419b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48228/0 (socket says 172.21.15.69:48228) 2023-07-12T15:46:14.125 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0d3fff700 1 -- 172.21.15.69:0/1762354255 learned_addr learned my addr 172.21.15.69:0/1762354255 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:46:14.125 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0d8f6d700 1 --2- 172.21.15.69:0/1762354255 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0d4104930 0x7fd0d4195750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:14.125 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0cbfff700 1 -- 172.21.15.69:0/1762354255 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0d4104930 msgr2=0x7fd0d4195750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:14.125 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0cbfff700 1 --2- 172.21.15.69:0/1762354255 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0d4104930 0x7fd0d4195750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:14.125 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0cbfff700 1 -- 172.21.15.69:0/1762354255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40ff2d0 msgr2=0x7fd0d419b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:14.125 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0cbfff700 1 --2- 172.21.15.69:0/1762354255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40ff2d0 0x7fd0d419b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:14.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0cbfff700 1 -- 172.21.15.69:0/1762354255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc009d00 con 0x7fd0d40ffc80 2023-07-12T15:46:14.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0d3fff700 1 --2- 172.21.15.69:0/1762354255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40ff2d0 0x7fd0d419b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:46:14.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0d8f6d700 1 --2- 172.21.15.69:0/1762354255 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0d4104930 0x7fd0d4195750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-12T15:46:14.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0cbfff700 1 --2- 172.21.15.69:0/1762354255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d40ffc80 0x7fd0d419bc20 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc00c970 tx=0x7fd0bc00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:46:14.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0d1ffb700 1 -- 172.21.15.69:0/1762354255 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc007b00 con 0x7fd0d40ffc80 2023-07-12T15:46:14.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0da9d0700 1 -- 172.21.15.69:0/1762354255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c4009520 con 0x7fd0d40ffc80 2023-07-12T15:46:14.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0d1ffb700 1 -- 172.21.15.69:0/1762354255 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7fd0bc00ce60 con 0x7fd0d40ffc80 2023-07-12T15:46:14.126 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0d1ffb700 1 -- 172.21.15.69:0/1762354255 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc0052b0 con 0x7fd0d40ffc80 2023-07-12T15:46:14.127 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.119+0000 7fd0da9d0700 1 -- 172.21.15.69:0/1762354255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d41a24f0 con 0x7fd0d40ffc80 2023-07-12T15:46:14.127 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.120+0000 7fd0da9d0700 1 -- 172.21.15.69:0/1762354255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d40feb20 con 0x7fd0d40ffc80 2023-07-12T15:46:14.128 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.121+0000 7fd0d1ffb700 1 -- 172.21.15.69:0/1762354255 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7fd0bc005450 con 0x7fd0d40ffc80 2023-07-12T15:46:14.128 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.121+0000 7fd0d1ffb700 1 --2- 172.21.15.69:0/1762354255 >> v2:172.21.15.69:6800/1242716914 conn(0x7fd0c006c810 0x7fd0c006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:14.128 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.121+0000 7fd0d1ffb700 1 -- 172.21.15.69:0/1762354255 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(768..768 src has 1..768) v4 ==== 8059+0+0 (secure 0 0 0) 0x7fd0bc08be50 con 0x7fd0d40ffc80 2023-07-12T15:46:14.128 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.121+0000 7fd0d3fff700 1 --2- 172.21.15.69:0/1762354255 >> v2:172.21.15.69:6800/1242716914 conn(0x7fd0c006c810 0x7fd0c006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:14.129 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.121+0000 7fd0d1ffb700 1 -- 172.21.15.69:0/1762354255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=769}) v3 -- 0x7fd0c00780c0 con 0x7fd0d40ffc80 2023-07-12T15:46:14.129 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.121+0000 7fd0d3fff700 1 --2- 172.21.15.69:0/1762354255 >> v2:172.21.15.69:6800/1242716914 conn(0x7fd0c006c810 0x7fd0c006ecc0 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400fae0 tx=0x7fd0c4019040 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:46:14.129 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.125+0000 7fd0d1ffb700 1 -- 172.21.15.69:0/1762354255 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd0bc01d060 con 0x7fd0d40ffc80 2023-07-12T15:46:14.318 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.316+0000 7fd0da9d0700 1 -- 172.21.15.69:0/1762354255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fd0d4196c60 con 0x7fd0d40ffc80 2023-07-12T15:46:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:13 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]': finished 2023-07-12T15:46:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:13 smithi145 ceph-mon[95801]: osdmap e767: 8 total, 8 up, 8 in 2023-07-12T15:46:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:13 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/2366416202' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:46:14.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:13 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:46:14.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:13 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:46:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:13 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]': finished 2023-07-12T15:46:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:13 smithi069 ceph-mon[102777]: osdmap e767: 8 total, 8 up, 8 in 2023-07-12T15:46:14.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:13 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/2366416202' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:46:14.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:13 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:46:14.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:13 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:46:14.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:13 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]': finished 2023-07-12T15:46:14.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:13 smithi069 ceph-mon[94241]: osdmap e767: 8 total, 8 up, 8 in 2023-07-12T15:46:14.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:13 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/2366416202' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:46:14.488 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:13 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]: dispatch 2023-07-12T15:46:14.489 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:13 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:46:14.998 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.996+0000 7fd0d1ffb700 1 -- 172.21.15.69:0/1762354255 <== mon.2 v2:172.21.15.69:3301/0 7 ==== osd_map(769..769 src has 1..769) v4 ==== 744+0+0 (secure 0 0 0) 0x7fd0bc005700 con 0x7fd0d40ffc80 2023-07-12T15:46:14.999 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:14.996+0000 7fd0d1ffb700 1 -- 172.21.15.69:0/1762354255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=770}) v3 -- 0x7fd0c0078c50 con 0x7fd0d40ffc80 2023-07-12T15:46:15.003 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:15.001+0000 7fd0d1ffb700 1 -- 172.21.15.69:0/1762354255 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v769) v1 ==== 221+0+0 (secure 0 0 0) 0x7fd0bc0595a0 con 0x7fd0d40ffc80 2023-07-12T15:46:15.127 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:15.125+0000 7fd0da9d0700 1 -- 172.21.15.69:0/1762354255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fd0d4067150 con 0x7fd0d40ffc80 2023-07-12T15:46:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]': finished 2023-07-12T15:46:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:14 smithi145 ceph-mon[95801]: osdmap e768: 8 total, 8 up, 8 in 2023-07-12T15:46:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:14 smithi145 ceph-mon[95801]: pgmap v1243: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:46:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:14 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1762354255' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:46:15.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:14 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:46:15.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]': finished 2023-07-12T15:46:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:14 smithi069 ceph-mon[102777]: osdmap e768: 8 total, 8 up, 8 in 2023-07-12T15:46:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:14 smithi069 ceph-mon[102777]: pgmap v1243: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:46:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:14 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1762354255' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:46:15.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:14 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:46:15.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_bytes", "val": "0"}]': finished 2023-07-12T15:46:15.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:14 smithi069 ceph-mon[94241]: osdmap e768: 8 total, 8 up, 8 in 2023-07-12T15:46:15.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:14 smithi069 ceph-mon[94241]: pgmap v1243: 188 pgs: 188 active+clean; 840 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:46:15.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:14 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1762354255' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:46:15.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:14 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:46:15.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:46:15 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:46:16.005 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.004+0000 7fd0d1ffb700 1 -- 172.21.15.69:0/1762354255 <== mon.2 v2:172.21.15.69:3301/0 9 ==== osd_map(770..770 src has 1..770) v4 ==== 744+0+0 (secure 0 0 0) 0x7fd0bc0597d0 con 0x7fd0d40ffc80 2023-07-12T15:46:16.006 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.004+0000 7fd0d1ffb700 1 -- 172.21.15.69:0/1762354255 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=771}) v3 -- 0x7fd0c0079080 con 0x7fd0d40ffc80 2023-07-12T15:46:16.009 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.008+0000 7fd0d1ffb700 1 -- 172.21.15.69:0/1762354255 <== mon.2 v2:172.21.15.69:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 v770) v1 ==== 221+0+0 (secure 0 0 0) 0x7fd0bc0595a0 con 0x7fd0d40ffc80 2023-07-12T15:46:16.012 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.011+0000 7fd0da9d0700 1 -- 172.21.15.69:0/1762354255 >> v2:172.21.15.69:6800/1242716914 conn(0x7fd0c006c810 msgr2=0x7fd0c006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:16.013 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.011+0000 7fd0da9d0700 1 --2- 172.21.15.69:0/1762354255 >> v2:172.21.15.69:6800/1242716914 conn(0x7fd0c006c810 0x7fd0c006ecc0 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400fae0 tx=0x7fd0c4019040 comp rx=0 tx=0).stop 2023-07-12T15:46:16.013 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.011+0000 7fd0da9d0700 1 -- 172.21.15.69:0/1762354255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d40ffc80 msgr2=0x7fd0d419bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:16.013 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.011+0000 7fd0da9d0700 1 --2- 172.21.15.69:0/1762354255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d40ffc80 0x7fd0d419bc20 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc00c970 tx=0x7fd0bc00cc80 comp rx=0 tx=0).stop 2023-07-12T15:46:16.013 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.011+0000 7fd0da9d0700 1 -- 172.21.15.69:0/1762354255 shutdown_connections 2023-07-12T15:46:16.013 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.011+0000 7fd0da9d0700 1 --2- 172.21.15.69:0/1762354255 >> v2:172.21.15.69:6800/1242716914 conn(0x7fd0c006c810 0x7fd0c006ecc0 unknown :-1 s=CLOSED pgs=3248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:16.014 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.011+0000 7fd0da9d0700 1 --2- 172.21.15.69:0/1762354255 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0d4104930 0x7fd0d4195750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:16.014 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.011+0000 7fd0da9d0700 1 --2- 172.21.15.69:0/1762354255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40ff2d0 0x7fd0d419b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:16.014 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.011+0000 7fd0da9d0700 1 --2- 172.21.15.69:0/1762354255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d40ffc80 0x7fd0d419bc20 unknown :-1 s=CLOSED pgs=2779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:16.014 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.011+0000 7fd0da9d0700 1 -- 172.21.15.69:0/1762354255 >> 172.21.15.69:0/1762354255 conn(0x7fd0d40748c0 msgr2=0x7fd0d410f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:46:16.014 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.011+0000 7fd0da9d0700 1 -- 172.21.15.69:0/1762354255 shutdown_connections 2023-07-12T15:46:16.014 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:16.011+0000 7fd0da9d0700 1 -- 172.21.15.69:0/1762354255 wait complete. 2023-07-12T15:46:16.015 INFO:tasks.workunit.client.0.smithi069.stderr:set-quota max_objects = 0 for pool e6659867-2dff-445e-9a1e-87d1f41b1a67 2023-07-12T15:46:16.024 INFO:tasks.workunit.client.0.smithi069.stderr:+ sleep 30 2023-07-12T15:46:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]': finished 2023-07-12T15:46:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:15 smithi145 ceph-mon[95801]: osdmap e769: 8 total, 8 up, 8 in 2023-07-12T15:46:16.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:15 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/1762354255' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:46:16.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:15 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:46:16.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:15 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:16.485 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:46:16] "GET /metrics HTTP/1.1" 200 228503 "" "Prometheus/2.43.0" 2023-07-12T15:46:16.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]': finished 2023-07-12T15:46:16.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:15 smithi069 ceph-mon[102777]: osdmap e769: 8 total, 8 up, 8 in 2023-07-12T15:46:16.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:15 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/1762354255' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:46:16.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:15 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:46:16.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:15 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]': finished 2023-07-12T15:46:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:15 smithi069 ceph-mon[94241]: osdmap e769: 8 total, 8 up, 8 in 2023-07-12T15:46:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:15 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/1762354255' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:46:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:15 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]: dispatch 2023-07-12T15:46:16.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:15 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:17.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:17 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]': finished 2023-07-12T15:46:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:17 smithi145 ceph-mon[95801]: osdmap e770: 8 total, 8 up, 8 in 2023-07-12T15:46:17.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:17 smithi145 ceph-mon[95801]: pgmap v1246: 188 pgs: 188 active+clean; 850 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2023-07-12T15:46:17.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:17 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]': finished 2023-07-12T15:46:17.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:17 smithi069 ceph-mon[102777]: osdmap e770: 8 total, 8 up, 8 in 2023-07-12T15:46:17.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:17 smithi069 ceph-mon[102777]: pgmap v1246: 188 pgs: 188 active+clean; 850 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2023-07-12T15:46:17.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:17 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "field": "max_objects", "val": "0"}]': finished 2023-07-12T15:46:17.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:17 smithi069 ceph-mon[94241]: osdmap e770: 8 total, 8 up, 8 in 2023-07-12T15:46:17.487 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:17 smithi069 ceph-mon[94241]: pgmap v1246: 188 pgs: 188 active+clean; 850 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2023-07-12T15:46:19.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:19 smithi145 ceph-mon[95801]: pgmap v1247: 188 pgs: 188 active+clean; 850 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 2.1 KiB/s wr, 1 op/s 2023-07-12T15:46:19.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:19 smithi069 ceph-mon[102777]: pgmap v1247: 188 pgs: 188 active+clean; 850 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 2.1 KiB/s wr, 1 op/s 2023-07-12T15:46:19.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:19 smithi069 ceph-mon[94241]: pgmap v1247: 188 pgs: 188 active+clean; 850 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 2.1 KiB/s wr, 1 op/s 2023-07-12T15:46:21.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:21 smithi145 ceph-mon[95801]: pgmap v1248: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.7 KiB/s wr, 1 op/s 2023-07-12T15:46:21.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:21 smithi069 ceph-mon[102777]: pgmap v1248: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.7 KiB/s wr, 1 op/s 2023-07-12T15:46:21.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:21 smithi069 ceph-mon[94241]: pgmap v1248: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.7 KiB/s wr, 1 op/s 2023-07-12T15:46:23.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:22 smithi145 ceph-mon[95801]: pgmap v1249: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 1 op/s 2023-07-12T15:46:23.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:22 smithi069 ceph-mon[94241]: pgmap v1249: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 1 op/s 2023-07-12T15:46:23.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:22 smithi069 ceph-mon[102777]: pgmap v1249: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 1 op/s 2023-07-12T15:46:25.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:46:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:46:25.601 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:25 smithi145 ceph-mon[95801]: pgmap v1250: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 551 B/s rd, 0 op/s 2023-07-12T15:46:25.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:25 smithi069 ceph-mon[102777]: pgmap v1250: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 551 B/s rd, 0 op/s 2023-07-12T15:46:25.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:25 smithi069 ceph-mon[94241]: pgmap v1250: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 551 B/s rd, 0 op/s 2023-07-12T15:46:26.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:26 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:26.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:26 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:26.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:26 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:46:26] "GET /metrics HTTP/1.1" 200 228503 "" "Prometheus/2.43.0" 2023-07-12T15:46:26.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:26 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:27.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:27 smithi069 ceph-mon[102777]: pgmap v1251: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 995 B/s rd, 0 op/s 2023-07-12T15:46:27.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:27 smithi069 ceph-mon[94241]: pgmap v1251: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 995 B/s rd, 0 op/s 2023-07-12T15:46:27.850 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:27 smithi145 ceph-mon[95801]: pgmap v1251: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 995 B/s rd, 0 op/s 2023-07-12T15:46:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:29 smithi069 ceph-mon[102777]: pgmap v1252: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:29.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:29 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:46:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:29 smithi069 ceph-mon[94241]: pgmap v1252: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:29.737 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:29 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:46:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:29 smithi145 ceph-mon[95801]: pgmap v1252: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:29.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:29 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:46:31.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:31 smithi069 ceph-mon[102777]: pgmap v1253: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:31.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:31 smithi069 ceph-mon[94241]: pgmap v1253: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:31.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:31 smithi145 ceph-mon[95801]: pgmap v1253: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:33.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:32 smithi145 ceph-mon[95801]: pgmap v1254: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:46:33.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:32 smithi069 ceph-mon[102777]: pgmap v1254: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:46:33.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:32 smithi069 ceph-mon[94241]: pgmap v1254: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-07-12T15:46:34.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:33 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:46:34.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:33 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:46:34.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:33 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-12T15:46:35.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:34 smithi145 ceph-mon[95801]: pgmap v1255: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:35.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:34 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:46:35.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:34 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:46:35.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:34 smithi145 ceph-mon[95801]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:46:35.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:34 smithi069 ceph-mon[94241]: pgmap v1255: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:35.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:34 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:46:35.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:34 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:46:35.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:34 smithi069 ceph-mon[94241]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:46:35.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:35 smithi069 ceph-mon[102777]: pgmap v1255: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:35.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:35 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-12T15:46:35.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:35 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-12T15:46:35.487 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:35 smithi069 ceph-mon[102777]: from='mgr.24524 ' entity='mgr.y' 2023-07-12T15:46:35.851 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:46:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:46:36.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:36 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:36.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:36 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:36.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:35 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:36.487 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:36 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:46:36] "GET /metrics HTTP/1.1" 200 228512 "" "Prometheus/2.43.0" 2023-07-12T15:46:37.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:37 smithi145 ceph-mon[95801]: pgmap v1256: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:37.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:37 smithi069 ceph-mon[102777]: pgmap v1256: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:37.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:37 smithi069 ceph-mon[94241]: pgmap v1256: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:39.600 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:39 smithi145 ceph-mon[95801]: pgmap v1257: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:39.736 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:39 smithi069 ceph-mon[102777]: pgmap v1257: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:39.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:39 smithi069 ceph-mon[94241]: pgmap v1257: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:41.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:41 smithi069 ceph-mon[102777]: pgmap v1258: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:41.736 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:41 smithi069 ceph-mon[94241]: pgmap v1258: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:41.851 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:41 smithi145 ceph-mon[95801]: pgmap v1258: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:43.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:42 smithi145 ceph-mon[95801]: pgmap v1259: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:43.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:42 smithi069 ceph-mon[102777]: pgmap v1259: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:43.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:42 smithi069 ceph-mon[94241]: pgmap v1259: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:44.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:43 smithi145 ceph-mon[95801]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:46:44.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:43 smithi069 ceph-mon[102777]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:46:44.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:43 smithi069 ceph-mon[94241]: from='mgr.24524 172.21.15.69:0/1747677394' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-07-12T15:46:45.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:44 smithi145 ceph-mon[95801]: pgmap v1260: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:45.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:44 smithi069 ceph-mon[102777]: pgmap v1260: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:45.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:44 smithi069 ceph-mon[94241]: pgmap v1260: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-07-12T15:46:45.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:46:45 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug there is no tcmu-runner data available 2023-07-12T15:46:46.026 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool delete e6659867-2dff-445e-9a1e-87d1f41b1a67 e6659867-2dff-445e-9a1e-87d1f41b1a67 --yes-i-really-really-mean-it 2023-07-12T15:46:46.138 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.137+0000 7f4575a9a700 1 -- 172.21.15.69:0/1805128761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4570069a20 msgr2=0x7f457010cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:46.139 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.137+0000 7f4575a9a700 1 --2- 172.21.15.69:0/1805128761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4570069a20 0x7f457010cf70 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f456800b0a0 tx=0x7f456800b3b0 comp rx=0 tx=0).stop 2023-07-12T15:46:46.139 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.137+0000 7f4575a9a700 1 -- 172.21.15.69:0/1805128761 shutdown_connections 2023-07-12T15:46:46.139 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.137+0000 7f4575a9a700 1 --2- 172.21.15.69:0/1805128761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4570069a20 0x7f457010cf70 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:46.139 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.137+0000 7f4575a9a700 1 --2- 172.21.15.69:0/1805128761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4570106140 0x7f4570106510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:46.139 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.137+0000 7f4575a9a700 1 --2- 172.21.15.69:0/1805128761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4570069090 0x7f45700694e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:46.139 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.137+0000 7f4575a9a700 1 -- 172.21.15.69:0/1805128761 >> 172.21.15.69:0/1805128761 conn(0x7f4570076f90 msgr2=0x7f4570077390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:46:46.140 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.137+0000 7f4575a9a700 1 -- 172.21.15.69:0/1805128761 shutdown_connections 2023-07-12T15:46:46.140 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.137+0000 7f4575a9a700 1 -- 172.21.15.69:0/1805128761 wait complete. 2023-07-12T15:46:46.142 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.137+0000 7f4575a9a700 1 Processor -- start 2023-07-12T15:46:46.142 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f4575a9a700 1 -- start start 2023-07-12T15:46:46.142 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f4575a9a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4570069090 0x7f45700ff080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:46.142 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f4575a9a700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4570069a20 0x7f45700ff5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:46.142 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f4575a9a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4570106140 0x7f45701a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:46.143 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f4575a9a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45700ffdf0 con 0x7f4570069090 2023-07-12T15:46:46.143 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f4575a9a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45701a4340 con 0x7f4570106140 2023-07-12T15:46:46.143 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f4575a9a700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45701a4480 con 0x7f4570069a20 2023-07-12T15:46:46.143 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f456f7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4570069090 0x7f45700ff080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:46.143 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f456f7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4570069090 0x7f45700ff080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48242/0 (socket says 172.21.15.69:48242) 2023-07-12T15:46:46.143 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f456effd700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4570069a20 0x7f45700ff5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:46.143 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f456f7fe700 1 -- 172.21.15.69:0/3704079220 learned_addr learned my addr 172.21.15.69:0/3704079220 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:46:46.144 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f456ffff700 1 --2- 172.21.15.69:0/3704079220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4570106140 0x7f45701a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:46.144 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f456f7fe700 1 -- 172.21.15.69:0/3704079220 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4570069a20 msgr2=0x7f45700ff5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:46.144 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f456f7fe700 1 --2- 172.21.15.69:0/3704079220 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4570069a20 0x7f45700ff5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:46.144 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f456f7fe700 1 -- 172.21.15.69:0/3704079220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4570106140 msgr2=0x7f45701a3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:46.144 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f456f7fe700 1 --2- 172.21.15.69:0/3704079220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4570106140 0x7f45701a3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:46.144 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f456f7fe700 1 -- 172.21.15.69:0/3704079220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4558009520 con 0x7f4570069090 2023-07-12T15:46:46.144 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.138+0000 7f456f7fe700 1 --2- 172.21.15.69:0/3704079220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4570069090 0x7f45700ff080 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f455800eb30 tx=0x7f455800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:46:46.145 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.139+0000 7f456cff9700 1 -- 172.21.15.69:0/3704079220 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558009880 con 0x7f4570069090 2023-07-12T15:46:46.145 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.139+0000 7f456cff9700 1 -- 172.21.15.69:0/3704079220 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f45580044b0 con 0x7f4570069090 2023-07-12T15:46:46.145 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.139+0000 7f4575a9a700 1 -- 172.21.15.69:0/3704079220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4568009d00 con 0x7f4570069090 2023-07-12T15:46:46.145 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.139+0000 7f456cff9700 1 -- 172.21.15.69:0/3704079220 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558003d80 con 0x7f4570069090 2023-07-12T15:46:46.145 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.139+0000 7f4575a9a700 1 -- 172.21.15.69:0/3704079220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45701a4a30 con 0x7f4570069090 2023-07-12T15:46:46.145 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.140+0000 7f4575a9a700 1 -- 172.21.15.69:0/3704079220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4570108650 con 0x7f4570069090 2023-07-12T15:46:46.145 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.140+0000 7f456cff9700 1 -- 172.21.15.69:0/3704079220 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f4558003ee0 con 0x7f4570069090 2023-07-12T15:46:46.146 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.140+0000 7f456cff9700 1 --2- 172.21.15.69:0/3704079220 >> v2:172.21.15.69:6800/1242716914 conn(0x7f455c06c810 0x7f455c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:46.146 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.141+0000 7f456cff9700 1 -- 172.21.15.69:0/3704079220 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(770..770 src has 1..770) v4 ==== 8059+0+0 (secure 0 0 0) 0x7f4558014070 con 0x7f4570069090 2023-07-12T15:46:46.146 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.141+0000 7f456effd700 1 --2- 172.21.15.69:0/3704079220 >> v2:172.21.15.69:6800/1242716914 conn(0x7f455c06c810 0x7f455c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:46.146 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.141+0000 7f456cff9700 1 -- 172.21.15.69:0/3704079220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=771}) v3 -- 0x7f455c0780c0 con 0x7f4570069090 2023-07-12T15:46:46.146 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.141+0000 7f456effd700 1 --2- 172.21.15.69:0/3704079220 >> v2:172.21.15.69:6800/1242716914 conn(0x7f455c06c810 0x7f455c06ecc0 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f4570100870 tx=0x7f456000a300 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:46:46.147 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.145+0000 7f456cff9700 1 -- 172.21.15.69:0/3704079220 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4558059ef0 con 0x7f4570069090 2023-07-12T15:46:46.339 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:46.338+0000 7f4575a9a700 1 -- 172.21.15.69:0/3704079220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pool2": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f45701005b0 con 0x7f4570069090 2023-07-12T15:46:46.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:45 smithi145 ceph-mon[95801]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:46.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:45 smithi069 ceph-mon[102777]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:46.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:45 smithi069 ceph-mon[94241]: from='client.24487 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-07-12T15:46:46.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:46 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: ::ffff:172.21.15.145 - - [12/Jul/2023:15:46:46] "GET /metrics HTTP/1.1" 200 228476 "" "Prometheus/2.43.0" 2023-07-12T15:46:47.012 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.010+0000 7f456cff9700 1 -- 172.21.15.69:0/3704079220 <== mon.0 v2:172.21.15.69:3300/0 7 ==== osd_map(771..771 src has 1..771) v4 ==== 412+0+0 (secure 0 0 0) 0x7f4558016330 con 0x7f4570069090 2023-07-12T15:46:47.013 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.011+0000 7f456cff9700 1 -- 172.21.15.69:0/3704079220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=772}) v3 -- 0x7f455c078970 con 0x7f4570069090 2023-07-12T15:46:47.018 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.016+0000 7f456cff9700 1 -- 172.21.15.69:0/3704079220 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pool2": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "yes_i_really_really_mean_it": true}]=0 pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' removed v771) v1 ==== 248+0+0 (secure 0 0 0) 0x7f4558059a80 con 0x7f4570069090 2023-07-12T15:46:47.148 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.146+0000 7f4575a9a700 1 -- 172.21.15.69:0/3704079220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pool2": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f45701081c0 con 0x7f4570069090 2023-07-12T15:46:47.148 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.147+0000 7f456cff9700 1 -- 172.21.15.69:0/3704079220 <== mon.0 v2:172.21.15.69:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pool2": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "yes_i_really_really_mean_it": true}]=0 pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' does not exist v771) v1 ==== 255+0+0 (secure 0 0 0) 0x7f4558059c60 con 0x7f4570069090 2023-07-12T15:46:47.150 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.149+0000 7f4575a9a700 1 -- 172.21.15.69:0/3704079220 >> v2:172.21.15.69:6800/1242716914 conn(0x7f455c06c810 msgr2=0x7f455c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:47.150 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.149+0000 7f4575a9a700 1 --2- 172.21.15.69:0/3704079220 >> v2:172.21.15.69:6800/1242716914 conn(0x7f455c06c810 0x7f455c06ecc0 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f4570100870 tx=0x7f456000a300 comp rx=0 tx=0).stop 2023-07-12T15:46:47.150 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.149+0000 7f4575a9a700 1 -- 172.21.15.69:0/3704079220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4570069090 msgr2=0x7f45700ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:47.151 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.149+0000 7f4575a9a700 1 --2- 172.21.15.69:0/3704079220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4570069090 0x7f45700ff080 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f455800eb30 tx=0x7f455800ee40 comp rx=0 tx=0).stop 2023-07-12T15:46:47.151 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.149+0000 7f4575a9a700 1 -- 172.21.15.69:0/3704079220 shutdown_connections 2023-07-12T15:46:47.151 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.149+0000 7f4575a9a700 1 --2- 172.21.15.69:0/3704079220 >> v2:172.21.15.69:6800/1242716914 conn(0x7f455c06c810 0x7f455c06ecc0 unknown :-1 s=CLOSED pgs=3249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:47.151 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.149+0000 7f4575a9a700 1 --2- 172.21.15.69:0/3704079220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4570106140 0x7f45701a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:47.151 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.149+0000 7f4575a9a700 1 --2- 172.21.15.69:0/3704079220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4570069090 0x7f45700ff080 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:47.151 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.149+0000 7f4575a9a700 1 --2- 172.21.15.69:0/3704079220 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4570069a20 0x7f45700ff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:47.151 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.149+0000 7f4575a9a700 1 -- 172.21.15.69:0/3704079220 >> 172.21.15.69:0/3704079220 conn(0x7f4570076f90 msgr2=0x7f45700fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:46:47.152 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.149+0000 7f4575a9a700 1 -- 172.21.15.69:0/3704079220 shutdown_connections 2023-07-12T15:46:47.152 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.149+0000 7f4575a9a700 1 -- 172.21.15.69:0/3704079220 wait complete. 2023-07-12T15:46:47.152 INFO:tasks.workunit.client.0.smithi069.stderr:pool 'e6659867-2dff-445e-9a1e-87d1f41b1a67' does not exist 2023-07-12T15:46:47.160 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph osd pool delete 4d559536-7353-4f04-abe6-10edbbd13922 4d559536-7353-4f04-abe6-10edbbd13922 --yes-i-really-really-mean-it 2023-07-12T15:46:47.263 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.261+0000 7f48891c8700 1 -- 172.21.15.69:0/882862728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48841042c0 msgr2=0x7f488410be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:47.263 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.261+0000 7f48891c8700 1 --2- 172.21.15.69:0/882862728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48841042c0 0x7f488410be90 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f487c0099c0 tx=0x7f487c009cd0 comp rx=0 tx=0).stop 2023-07-12T15:46:47.263 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.261+0000 7f48891c8700 1 -- 172.21.15.69:0/882862728 shutdown_connections 2023-07-12T15:46:47.264 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.261+0000 7f48891c8700 1 --2- 172.21.15.69:0/882862728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48841042c0 0x7f488410be90 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:47.264 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.261+0000 7f48891c8700 1 --2- 172.21.15.69:0/882862728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48840ff280 0x7f48840ff650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:47.264 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.261+0000 7f48891c8700 1 --2- 172.21.15.69:0/882862728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48840ffc20 0x7f4884103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:47.264 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.261+0000 7f48891c8700 1 -- 172.21.15.69:0/882862728 >> 172.21.15.69:0/882862728 conn(0x7f48840faa90 msgr2=0x7f48840fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:46:47.264 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.261+0000 7f48891c8700 1 -- 172.21.15.69:0/882862728 shutdown_connections 2023-07-12T15:46:47.264 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.261+0000 7f48891c8700 1 -- 172.21.15.69:0/882862728 wait complete. 2023-07-12T15:46:47.266 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.262+0000 7f48891c8700 1 Processor -- start 2023-07-12T15:46:47.266 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.262+0000 7f48891c8700 1 -- start start 2023-07-12T15:46:47.266 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.262+0000 7f48891c8700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48840ff280 0x7f4884194450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:47.267 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.262+0000 7f48891c8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48840ffc20 0x7f4884194990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:47.267 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.262+0000 7f48891c8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48841042c0 0x7f4884194ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:47.267 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.262+0000 7f48891c8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48841a3c90 con 0x7f48840ffc20 2023-07-12T15:46:47.267 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.262+0000 7f48891c8700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48841a3e00 con 0x7f48841042c0 2023-07-12T15:46:47.267 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.262+0000 7f48891c8700 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48841a3fc0 con 0x7f48840ff280 2023-07-12T15:46:47.267 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.262+0000 7f4882d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48840ff280 0x7f4884194450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:47.267 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.262+0000 7f488259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48840ffc20 0x7f4884194990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:47.267 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.262+0000 7f4882d9d700 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48840ff280 0x7f4884194450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:41684/0 (socket says 172.21.15.69:41684) 2023-07-12T15:46:47.268 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.263+0000 7f4882d9d700 1 -- 172.21.15.69:0/475789603 learned_addr learned my addr 172.21.15.69:0/475789603 (peer_addr_for_me v2:172.21.15.69:0/0) 2023-07-12T15:46:47.268 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.262+0000 7f488259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48840ffc20 0x7f4884194990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48258/0 (socket says 172.21.15.69:48258) 2023-07-12T15:46:47.268 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.263+0000 7f4882d9d700 1 -- 172.21.15.69:0/475789603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48841042c0 msgr2=0x7f4884194ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:47.268 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.263+0000 7f4882d9d700 1 --2- 172.21.15.69:0/475789603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48841042c0 0x7f4884194ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:47.268 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.263+0000 7f4882d9d700 1 -- 172.21.15.69:0/475789603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48840ffc20 msgr2=0x7f4884194990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:47.268 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.263+0000 7f4882d9d700 1 --2- 172.21.15.69:0/475789603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48840ffc20 0x7f4884194990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:47.268 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.263+0000 7f4882d9d700 1 -- 172.21.15.69:0/475789603 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c009520 con 0x7f48840ff280 2023-07-12T15:46:47.269 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.263+0000 7f488259c700 1 --2- 172.21.15.69:0/475789603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48840ffc20 0x7f4884194990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-12T15:46:47.269 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.263+0000 7f4882d9d700 1 --2- 172.21.15.69:0/475789603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48840ff280 0x7f4884194450 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f4874007e20 tx=0x7f487400a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:46:47.269 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.263+0000 7f487bfff700 1 -- 172.21.15.69:0/475789603 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487400ad40 con 0x7f48840ff280 2023-07-12T15:46:47.269 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.263+0000 7f48891c8700 1 -- 172.21.15.69:0/475789603 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487c0095f0 con 0x7f48840ff280 2023-07-12T15:46:47.269 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.263+0000 7f487bfff700 1 -- 172.21.15.69:0/475789603 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(25 keys) v1 ==== 1058+0+0 (secure 0 0 0) 0x7f48740128b0 con 0x7f48840ff280 2023-07-12T15:46:47.269 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.263+0000 7f487bfff700 1 -- 172.21.15.69:0/475789603 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48740116c0 con 0x7f48840ff280 2023-07-12T15:46:47.269 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.263+0000 7f48891c8700 1 -- 172.21.15.69:0/475789603 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48841a4620 con 0x7f48840ff280 2023-07-12T15:46:47.269 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.264+0000 7f48891c8700 1 -- 172.21.15.69:0/475789603 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4884067e30 con 0x7f48840ff280 2023-07-12T15:46:47.270 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.265+0000 7f487bfff700 1 -- 172.21.15.69:0/475789603 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 20) v1 ==== 89841+0+0 (secure 0 0 0) 0x7f4874012420 con 0x7f48840ff280 2023-07-12T15:46:47.271 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.265+0000 7f487bfff700 1 --2- 172.21.15.69:0/475789603 >> v2:172.21.15.69:6800/1242716914 conn(0x7f487006c810 0x7f487006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-12T15:46:47.271 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.266+0000 7f487bfff700 1 -- 172.21.15.69:0/475789603 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(771..771 src has 1..771) v4 ==== 7671+0+0 (secure 0 0 0) 0x7f487408c660 con 0x7f48840ff280 2023-07-12T15:46:47.271 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.266+0000 7f488259c700 1 --2- 172.21.15.69:0/475789603 >> v2:172.21.15.69:6800/1242716914 conn(0x7f487006c810 0x7f487006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-12T15:46:47.271 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.266+0000 7f487bfff700 1 -- 172.21.15.69:0/475789603 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=772}) v3 -- 0x7f48700778a0 con 0x7f48840ff280 2023-07-12T15:46:47.271 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.266+0000 7f488259c700 1 --2- 172.21.15.69:0/475789603 >> v2:172.21.15.69:6800/1242716914 conn(0x7f487006c810 0x7f487006ecc0 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7f48840698b0 tx=0x7f486c009260 comp rx=0 tx=0).ready entity=mgr.24524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-12T15:46:47.271 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.269+0000 7f487bfff700 1 -- 172.21.15.69:0/475789603 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4874059fb0 con 0x7f48840ff280 2023-07-12T15:46:47.350 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:47 smithi145 ceph-mon[95801]: pgmap v1261: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:47.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:47 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3704079220' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pool2": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:47.458 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:47.456+0000 7f48891c8700 1 -- 172.21.15.69:0/475789603 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4884069540 con 0x7f48840ff280 2023-07-12T15:46:47.485 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:47 smithi069 ceph-mon[102777]: pgmap v1261: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:47.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:47 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3704079220' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pool2": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:47 smithi069 ceph-mon[94241]: pgmap v1261: 188 pgs: 188 active+clean; 850 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-07-12T15:46:47.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:47 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3704079220' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pool2": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:48.021 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.019+0000 7f487bfff700 1 -- 172.21.15.69:0/475789603 <== mon.2 v2:172.21.15.69:3301/0 7 ==== osd_map(772..772 src has 1..772) v4 ==== 412+0+0 (secure 0 0 0) 0x7f48740126d0 con 0x7f48840ff280 2023-07-12T15:46:48.021 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.020+0000 7f487bfff700 1 -- 172.21.15.69:0/475789603 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=773}) v3 -- 0x7f4870078140 con 0x7f48840ff280 2023-07-12T15:46:48.026 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.024+0000 7f487bfff700 1 -- 172.21.15.69:0/475789603 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]=0 pool '4d559536-7353-4f04-abe6-10edbbd13922' removed v772) v1 ==== 248+0+0 (secure 0 0 0) 0x7f4874016070 con 0x7f48840ff280 2023-07-12T15:46:48.154 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.153+0000 7f48891c8700 1 -- 172.21.15.69:0/475789603 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f48841a48a0 con 0x7f48840ff280 2023-07-12T15:46:48.154 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.153+0000 7f487bfff700 1 -- 172.21.15.69:0/475789603 <== mon.2 v2:172.21.15.69:3301/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]=0 pool '4d559536-7353-4f04-abe6-10edbbd13922' does not exist v772) v1 ==== 255+0+0 (secure 0 0 0) 0x7f4874059b40 con 0x7f48840ff280 2023-07-12T15:46:48.157 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.155+0000 7f48891c8700 1 -- 172.21.15.69:0/475789603 >> v2:172.21.15.69:6800/1242716914 conn(0x7f487006c810 msgr2=0x7f487006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:48.157 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.155+0000 7f48891c8700 1 --2- 172.21.15.69:0/475789603 >> v2:172.21.15.69:6800/1242716914 conn(0x7f487006c810 0x7f487006ecc0 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7f48840698b0 tx=0x7f486c009260 comp rx=0 tx=0).stop 2023-07-12T15:46:48.157 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.155+0000 7f48891c8700 1 -- 172.21.15.69:0/475789603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48840ff280 msgr2=0x7f4884194450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-12T15:46:48.157 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.155+0000 7f48891c8700 1 --2- 172.21.15.69:0/475789603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48840ff280 0x7f4884194450 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f4874007e20 tx=0x7f487400a340 comp rx=0 tx=0).stop 2023-07-12T15:46:48.157 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.155+0000 7f48891c8700 1 -- 172.21.15.69:0/475789603 shutdown_connections 2023-07-12T15:46:48.158 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.156+0000 7f48891c8700 1 --2- 172.21.15.69:0/475789603 >> v2:172.21.15.69:6800/1242716914 conn(0x7f487006c810 0x7f487006ecc0 unknown :-1 s=CLOSED pgs=3250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:48.158 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.156+0000 7f48891c8700 1 --2- 172.21.15.69:0/475789603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48841042c0 0x7f4884194ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:48.158 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.156+0000 7f48891c8700 1 --2- 172.21.15.69:0/475789603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48840ff280 0x7f4884194450 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:48.158 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.156+0000 7f48891c8700 1 --2- 172.21.15.69:0/475789603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48840ffc20 0x7f4884194990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-12T15:46:48.158 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.156+0000 7f48891c8700 1 -- 172.21.15.69:0/475789603 >> 172.21.15.69:0/475789603 conn(0x7f48840faa90 msgr2=0x7f48840fb410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-12T15:46:48.158 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.156+0000 7f48891c8700 1 -- 172.21.15.69:0/475789603 shutdown_connections 2023-07-12T15:46:48.158 INFO:tasks.workunit.client.0.smithi069.stderr:2023-07-12T15:46:48.156+0000 7f48891c8700 1 -- 172.21.15.69:0/475789603 wait complete. 2023-07-12T15:46:48.159 INFO:tasks.workunit.client.0.smithi069.stderr:pool '4d559536-7353-4f04-abe6-10edbbd13922' does not exist 2023-07-12T15:46:48.167 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo OK 2023-07-12T15:46:48.169 INFO:tasks.workunit.client.0.smithi069.stdout:OK 2023-07-12T15:46:48.169 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-07-12T15:46:48.169 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-07-12T15:46:48.239 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2023-07-12T15:46:48.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-07-12T15:46:48.304 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3704079220' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pool2": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "yes_i_really_really_mean_it": true}]': finished 2023-07-12T15:46:48.304 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:48 smithi069 ceph-mon[102777]: osdmap e771: 8 total, 8 up, 8 in 2023-07-12T15:46:48.304 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/3704079220' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pool2": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:48.304 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:48 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/475789603' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:48.305 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:48 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:48.305 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3704079220' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pool2": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "yes_i_really_really_mean_it": true}]': finished 2023-07-12T15:46:48.305 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:48 smithi069 ceph-mon[94241]: osdmap e771: 8 total, 8 up, 8 in 2023-07-12T15:46:48.305 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/3704079220' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pool2": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:48.305 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:48 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/475789603' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:48.305 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:48 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3704079220' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pool2": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "yes_i_really_really_mean_it": true}]': finished 2023-07-12T15:46:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:48 smithi145 ceph-mon[95801]: osdmap e771: 8 total, 8 up, 8 in 2023-07-12T15:46:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/3704079220' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "pool2": "e6659867-2dff-445e-9a1e-87d1f41b1a67", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:48.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:48 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/475789603' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:48.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:48 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:48.526 DEBUG:teuthology.parallel:result is None 2023-07-12T15:46:48.527 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-07-12T15:46:48.593 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-07-12T15:46:48.593 DEBUG:teuthology.orchestra.run.smithi069:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-07-12T15:46:48.650 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-07-12T15:46:48.650 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-12T15:46:48.661 INFO:tasks.cephadm:Teardown begin 2023-07-12T15:46:48.662 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-12T15:46:48.716 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-12T15:46:48.765 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-12T15:46:48.766 DEBUG:teuthology.orchestra.run.smithi069:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-12T15:46:48.783 DEBUG:teuthology.orchestra.run.smithi145:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-12T15:46:48.802 INFO:tasks.cephadm:Stopping all daemons... 2023-07-12T15:46:48.802 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-12T15:46:48.803 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.a 2023-07-12T15:46:49.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:48 smithi069 systemd[1]: Stopping Ceph mon.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:46:49.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-mon[94241]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2023-07-12T15:46:49.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]': finished 2023-07-12T15:46:49.174 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-mon[94241]: osdmap e772: 8 total, 8 up, 8 in 2023-07-12T15:46:49.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-mon[94241]: from='client.? 172.21.15.69:0/475789603' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:49.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-mon[94241]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:49.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-mon[94241]: pgmap v1264: 164 pgs: 164 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:46:49.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-mon[102777]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2023-07-12T15:46:49.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]': finished 2023-07-12T15:46:49.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-mon[102777]: osdmap e772: 8 total, 8 up, 8 in 2023-07-12T15:46:49.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-mon[102777]: from='client.? 172.21.15.69:0/475789603' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:49.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-mon[102777]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:49.176 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-mon[102777]: pgmap v1264: 164 pgs: 164 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:46:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:49 smithi145 ceph-mon[95801]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2023-07-12T15:46:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:49 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]': finished 2023-07-12T15:46:49.351 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:49 smithi145 ceph-mon[95801]: osdmap e772: 8 total, 8 up, 8 in 2023-07-12T15:46:49.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:49 smithi145 ceph-mon[95801]: from='client.? 172.21.15.69:0/475789603' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:49.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:49 smithi145 ceph-mon[95801]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4d559536-7353-4f04-abe6-10edbbd13922", "pool2": "4d559536-7353-4f04-abe6-10edbbd13922", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-12T15:46:49.352 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:49 smithi145 ceph-mon[95801]: pgmap v1264: 164 pgs: 164 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-07-12T15:46:49.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mon-a[94216]: 2023-07-12T15:46:49.170+0000 7efff8ff5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:46:49.486 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:49 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mon-a[94216]: 2023-07-12T15:46:49.170+0000 7efff8ff5700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-12T15:46:49.799 INFO:journalctl@ceph.mon.a.smithi069.stdout:Jul 12 15:46:49 smithi069 bash[205376]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mon-a 2023-07-12T15:46:49.806 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.a.service' 2023-07-12T15:46:49.845 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:46:49.846 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-12T15:46:49.846 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2023-07-12T15:46:49.846 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.c 2023-07-12T15:46:50.227 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:49 smithi069 systemd[1]: Stopping Ceph mon.c for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:46:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:50 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mon-c[102753]: 2023-07-12T15:46:50.224+0000 7f25f53fb700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:46:50.486 INFO:journalctl@ceph.mon.c.smithi069.stdout:Jul 12 15:46:50 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mon-c[102753]: 2023-07-12T15:46:50.224+0000 7f25f53fb700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2023-07-12T15:46:50.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:50 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:46:50] ENGINE Bus STOPPING 2023-07-12T15:46:50.486 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:50 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:46:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2023-07-12T15:46:50.487 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:50 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:46:50] ENGINE Bus STOPPED 2023-07-12T15:46:50.487 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:50 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:46:50] ENGINE Bus STARTING 2023-07-12T15:46:50.487 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:50 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:46:50] ENGINE Serving on http://:::9283 2023-07-12T15:46:50.487 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:50 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: [12/Jul/2023:15:46:50] ENGINE Bus STARTED 2023-07-12T15:46:50.863 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.c.service' 2023-07-12T15:46:50.899 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:46:50.900 INFO:tasks.cephadm.mon.b:Stopped mon.c 2023-07-12T15:46:50.900 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2023-07-12T15:46:50.900 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.b 2023-07-12T15:46:51.266 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:50 smithi145 systemd[1]: Stopping Ceph mon.b for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:46:51.535 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:51 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mon-b[95776]: 2023-07-12T15:46:51.263+0000 7f0bb0eef700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:46:51.535 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:51 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mon-b[95776]: 2023-07-12T15:46:51.263+0000 7f0bb0eef700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2023-07-12T15:46:51.812 INFO:journalctl@ceph.mon.b.smithi145.stdout:Jul 12 15:46:51 smithi145 bash[128629]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mon-b 2023-07-12T15:46:52.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mon.b.service' 2023-07-12T15:46:52.050 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:46:52.051 INFO:tasks.cephadm.mon.b:Stopped mon.b 2023-07-12T15:46:52.051 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-12T15:46:52.052 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mgr.y 2023-07-12T15:46:52.394 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:52 smithi069 systemd[1]: Stopping Ceph mgr.y for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:46:52.703 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:52 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:46:52.390+0000 7f40b2752700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:46:52.703 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:52 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y[94488]: 2023-07-12T15:46:52.390+0000 7f40b2752700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-07-12T15:46:52.946 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mgr.y.service' 2023-07-12T15:46:52.970 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:52 smithi069 bash[205675]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-y 2023-07-12T15:46:52.971 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:52 smithi069 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mgr.y.service: Succeeded. 2023-07-12T15:46:52.971 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Jul 12 15:46:52 smithi069 systemd[1]: Stopped Ceph mgr.y for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:46:52.983 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:46:52.983 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-12T15:46:52.984 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-12T15:46:52.984 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mgr.x 2023-07-12T15:46:53.351 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:46:53 smithi145 systemd[1]: Stopping Ceph mgr.x for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:46:53.695 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:46:53 smithi145 bash[128774]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-mgr-x 2023-07-12T15:46:53.876 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Jul 12 15:46:53 smithi145 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mgr.x.service: Main process exited, code=exited, status=143/n/a 2023-07-12T15:46:53.880 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@mgr.x.service' 2023-07-12T15:46:53.918 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:46:53.918 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-12T15:46:53.918 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-12T15:46:53.919 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.0 2023-07-12T15:46:54.235 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:46:53 smithi069 systemd[1]: Stopping Ceph osd.0 for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:46:54.735 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:46:54 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-0[107243]: 2023-07-12T15:46:54.294+0000 7f5ebde27700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:46:54.736 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:46:54 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-0[107243]: 2023-07-12T15:46:54.294+0000 7f5ebde27700 -1 osd.0 772 *** Got signal Terminated *** 2023-07-12T15:46:54.736 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:46:54 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-0[107243]: 2023-07-12T15:46:54.294+0000 7f5ebde27700 -1 osd.0 772 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-12T15:47:00.486 INFO:journalctl@ceph.osd.0.smithi069.stdout:Jul 12 15:47:00 smithi069 bash[205817]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-0 2023-07-12T15:47:00.487 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:00 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:00.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi069.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2023-07-12T15:47:00.487 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:00 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:00.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2023-07-12T15:47:01.682 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.0.service' 2023-07-12T15:47:01.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:47:01.721 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-12T15:47:01.721 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-12T15:47:01.722 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.1 2023-07-12T15:47:01.986 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:47:01 smithi069 systemd[1]: Stopping Ceph osd.1 for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:47:02.486 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:47:02 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-1[111786]: 2023-07-12T15:47:02.109+0000 7fab5a433700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:47:02.486 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:47:02 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-1[111786]: 2023-07-12T15:47:02.109+0000 7fab5a433700 -1 osd.1 772 *** Got signal Terminated *** 2023-07-12T15:47:02.486 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:47:02 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-1[111786]: 2023-07-12T15:47:02.109+0000 7fab5a433700 -1 osd.1 772 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-12T15:47:06.601 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:47:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:47:06.202Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.69:8765: connect: connection refused" 2023-07-12T15:47:06.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:47:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:47:06.202Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.69:8765: connect: connection refused" 2023-07-12T15:47:06.602 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:47:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:47:06.203Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.69:8765: connect: connection refused" 2023-07-12T15:47:06.603 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:47:06 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:47:06.204Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.69:8765: connect: connection refused" 2023-07-12T15:47:07.986 INFO:journalctl@ceph.osd.1.smithi069.stdout:Jul 12 15:47:07 smithi069 bash[206085]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-1 2023-07-12T15:47:08.950 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.1.service' 2023-07-12T15:47:08.990 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:47:08.990 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-12T15:47:08.991 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-12T15:47:08.991 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.2 2023-07-12T15:47:09.236 INFO:journalctl@ceph.osd.2.smithi069.stdout:Jul 12 15:47:09 smithi069 systemd[1]: Stopping Ceph osd.2 for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:47:09.735 INFO:journalctl@ceph.osd.2.smithi069.stdout:Jul 12 15:47:09 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-2[116331]: 2023-07-12T15:47:09.366+0000 7f00d21fd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:47:09.736 INFO:journalctl@ceph.osd.2.smithi069.stdout:Jul 12 15:47:09 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-2[116331]: 2023-07-12T15:47:09.366+0000 7f00d21fd700 -1 osd.2 772 *** Got signal Terminated *** 2023-07-12T15:47:09.736 INFO:journalctl@ceph.osd.2.smithi069.stdout:Jul 12 15:47:09 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-2[116331]: 2023-07-12T15:47:09.366+0000 7f00d21fd700 -1 osd.2 772 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-12T15:47:10.485 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:10 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:10.145Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi069.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2023-07-12T15:47:10.486 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:10 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:10.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi069.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2023-07-12T15:47:10.486 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:10 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:10.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2023-07-12T15:47:15.155 INFO:journalctl@ceph.osd.2.smithi069.stdout:Jul 12 15:47:15 smithi069 bash[206352]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-2 2023-07-12T15:47:16.522 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.2.service' 2023-07-12T15:47:16.562 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:47:16.562 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-12T15:47:16.562 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-12T15:47:16.563 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.3 2023-07-12T15:47:16.942 INFO:journalctl@ceph.osd.3.smithi069.stdout:Jul 12 15:47:16 smithi069 systemd[1]: Stopping Ceph osd.3 for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:47:17.235 INFO:journalctl@ceph.osd.3.smithi069.stdout:Jul 12 15:47:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-3[120895]: 2023-07-12T15:47:16.940+0000 7f9a28ff0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:47:17.236 INFO:journalctl@ceph.osd.3.smithi069.stdout:Jul 12 15:47:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-3[120895]: 2023-07-12T15:47:16.940+0000 7f9a28ff0700 -1 osd.3 772 *** Got signal Terminated *** 2023-07-12T15:47:17.236 INFO:journalctl@ceph.osd.3.smithi069.stdout:Jul 12 15:47:16 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-3[120895]: 2023-07-12T15:47:16.940+0000 7f9a28ff0700 -1 osd.3 772 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-12T15:47:20.490 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:20 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:20.146Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi069.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2023-07-12T15:47:20.490 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:20 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:20.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi069.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2023-07-12T15:47:20.490 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:20 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:20.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2023-07-12T15:47:21.101 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:20 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:20.796+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:22.100 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:21 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:21.819+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:22.235 INFO:journalctl@ceph.osd.3.smithi069.stdout:Jul 12 15:47:21 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-3[120895]: 2023-07-12T15:47:21.764+0000 7f9a21ddf700 -1 osd.3 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.148668+0000 front 2023-07-12T15:46:57.148844+0000 (oldest deadline 2023-07-12T15:47:21.749405+0000) 2023-07-12T15:47:22.725 INFO:journalctl@ceph.osd.3.smithi069.stdout:Jul 12 15:47:22 smithi069 bash[206614]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-3 2023-07-12T15:47:23.100 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:47:22 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:47:22.638+0000 7f55d26a7700 -1 osd.4 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.437484+0000 front 2023-07-12T15:46:57.437628+0000 (oldest deadline 2023-07-12T15:47:22.137557+0000) 2023-07-12T15:47:23.101 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:22 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:22.865+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:23.601 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:23 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:23.135+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:24.091 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.3.service' 2023-07-12T15:47:24.101 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:47:23 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:47:23.668+0000 7f55d26a7700 -1 osd.4 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.437484+0000 front 2023-07-12T15:46:57.437628+0000 (oldest deadline 2023-07-12T15:47:22.137557+0000) 2023-07-12T15:47:24.101 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:23 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:23.897+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:24.132 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:47:24.133 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-12T15:47:24.133 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-12T15:47:24.133 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.4 2023-07-12T15:47:24.492 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:47:24 smithi145 systemd[1]: Stopping Ceph osd.4 for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:47:24.493 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:24.123+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:24.850 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:47:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:47:24.490+0000 7f55d60b1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:47:24.851 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:47:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:47:24.490+0000 7f55d60b1700 -1 osd.4 772 *** Got signal Terminated *** 2023-07-12T15:47:24.851 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:47:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:47:24.490+0000 7f55d60b1700 -1 osd.4 772 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-12T15:47:24.851 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:47:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:47:24.631+0000 7f55d26a7700 -1 osd.4 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.437484+0000 front 2023-07-12T15:46:57.437628+0000 (oldest deadline 2023-07-12T15:47:22.137557+0000) 2023-07-12T15:47:25.161 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:24 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:24.863+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:25.449 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:25.159+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:25.851 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:47:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:47:25.593+0000 7f55d26a7700 -1 osd.4 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.437484+0000 front 2023-07-12T15:46:57.437628+0000 (oldest deadline 2023-07-12T15:47:22.137557+0000) 2023-07-12T15:47:25.851 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:25.447+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:58.659592+0000 front 2023-07-12T15:46:58.659422+0000 (oldest deadline 2023-07-12T15:47:24.559496+0000) 2023-07-12T15:47:26.124 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:25 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:25.871+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:26.452 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:26 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:26.123+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:26.850 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:47:26 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:47:26.587+0000 7f55d26a7700 -1 osd.4 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.437484+0000 front 2023-07-12T15:46:57.437628+0000 (oldest deadline 2023-07-12T15:47:22.137557+0000) 2023-07-12T15:47:26.851 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:26 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:26.450+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:58.659592+0000 front 2023-07-12T15:46:58.659422+0000 (oldest deadline 2023-07-12T15:47:24.559496+0000) 2023-07-12T15:47:27.350 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:26 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:26.879+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:27.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:27 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:27.098+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:27.850 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:27 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:27.471+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:58.659592+0000 front 2023-07-12T15:46:58.659422+0000 (oldest deadline 2023-07-12T15:47:24.559496+0000) 2023-07-12T15:47:27.851 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:47:27 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:47:27.572+0000 7f55d26a7700 -1 osd.4 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.437484+0000 front 2023-07-12T15:46:57.437628+0000 (oldest deadline 2023-07-12T15:47:22.137557+0000) 2023-07-12T15:47:28.139 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:27 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:27.865+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:28.514 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:28 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:28.137+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:28.851 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:28 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:28.512+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:58.659592+0000 front 2023-07-12T15:46:58.659422+0000 (oldest deadline 2023-07-12T15:47:24.559496+0000) 2023-07-12T15:47:28.851 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:28 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:28.512+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.560245+0000 front 2023-07-12T15:47:04.560068+0000 (oldest deadline 2023-07-12T15:47:28.060141+0000) 2023-07-12T15:47:28.851 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:47:28 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:47:28.574+0000 7f55d26a7700 -1 osd.4 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.437484+0000 front 2023-07-12T15:46:57.437628+0000 (oldest deadline 2023-07-12T15:47:22.137557+0000) 2023-07-12T15:47:28.852 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:47:28 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4[101534]: 2023-07-12T15:47:28.574+0000 7f55d26a7700 -1 osd.4 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:02.638631+0000 front 2023-07-12T15:47:02.638769+0000 (oldest deadline 2023-07-12T15:47:27.938592+0000) 2023-07-12T15:47:29.132 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:28 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:28.853+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:29.563 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:29 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:29.131+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:29.841 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:29 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:29.562+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:58.659592+0000 front 2023-07-12T15:46:58.659422+0000 (oldest deadline 2023-07-12T15:47:24.559496+0000) 2023-07-12T15:47:29.841 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:29 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:29.562+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.560245+0000 front 2023-07-12T15:47:04.560068+0000 (oldest deadline 2023-07-12T15:47:28.060141+0000) 2023-07-12T15:47:30.101 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:29 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:29.839+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:30.101 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:29 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:29.839+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:30.486 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:30 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:30.147Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi069.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2023-07-12T15:47:30.486 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:30 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:30.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi069.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2023-07-12T15:47:30.486 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:30 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:30.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2023-07-12T15:47:30.489 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:30 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:30.178+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:30.850 INFO:journalctl@ceph.osd.4.smithi145.stdout:Jul 12 15:47:30 smithi145 bash[128930]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-4 2023-07-12T15:47:30.851 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:30 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:30.561+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:58.659592+0000 front 2023-07-12T15:46:58.659422+0000 (oldest deadline 2023-07-12T15:47:24.559496+0000) 2023-07-12T15:47:30.851 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:30 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:30.561+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.560245+0000 front 2023-07-12T15:47:04.560068+0000 (oldest deadline 2023-07-12T15:47:28.060141+0000) 2023-07-12T15:47:31.134 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:30 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:30.877+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:31.134 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:30 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:30.877+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:31.521 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:31 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:31.132+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:31.521 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:31 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:31.132+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:31.839 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:31 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:31.554+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:58.659592+0000 front 2023-07-12T15:46:58.659422+0000 (oldest deadline 2023-07-12T15:47:24.559496+0000) 2023-07-12T15:47:31.839 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:31 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:31.554+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.560245+0000 front 2023-07-12T15:47:04.560068+0000 (oldest deadline 2023-07-12T15:47:28.060141+0000) 2023-07-12T15:47:31.846 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.4.service' 2023-07-12T15:47:31.886 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:47:31.886 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-12T15:47:31.887 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-12T15:47:31.887 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.5 2023-07-12T15:47:32.098 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:31 smithi145 systemd[1]: Stopping Ceph osd.5 for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:47:32.098 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:31 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:31.878+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:32.099 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:31 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:31.878+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:32.350 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:32.096+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:32.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:32.096+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:32.351 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:32.252+0000 7fa25e2ed700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:47:32.351 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:32.252+0000 7fa25e2ed700 -1 osd.5 772 *** Got signal Terminated *** 2023-07-12T15:47:32.351 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:32.252+0000 7fa25e2ed700 -1 osd.5 772 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-12T15:47:32.850 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:32.545+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:58.659592+0000 front 2023-07-12T15:46:58.659422+0000 (oldest deadline 2023-07-12T15:47:24.559496+0000) 2023-07-12T15:47:32.851 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:32.545+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.560245+0000 front 2023-07-12T15:47:04.560068+0000 (oldest deadline 2023-07-12T15:47:28.060141+0000) 2023-07-12T15:47:33.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:33.121+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:33.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:33.121+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:33.352 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:32.900+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:33.352 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:32 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:32.900+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:33.851 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:33.567+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:58.659592+0000 front 2023-07-12T15:46:58.659422+0000 (oldest deadline 2023-07-12T15:47:24.559496+0000) 2023-07-12T15:47:33.851 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:33.567+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.560245+0000 front 2023-07-12T15:47:04.560068+0000 (oldest deadline 2023-07-12T15:47:28.060141+0000) 2023-07-12T15:47:34.170 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:33.853+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:34.170 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:33 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:33.853+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:34.532 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:34.168+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:34.533 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:34.168+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:34.850 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:34.531+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:58.659592+0000 front 2023-07-12T15:46:58.659422+0000 (oldest deadline 2023-07-12T15:47:24.559496+0000) 2023-07-12T15:47:34.851 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:34.531+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.560245+0000 front 2023-07-12T15:47:04.560068+0000 (oldest deadline 2023-07-12T15:47:28.060141+0000) 2023-07-12T15:47:35.131 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:34.864+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:35.131 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:34 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:34.864+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:35.516 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:35.129+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:35.516 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:35.129+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:35.850 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:35.514+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:58.659592+0000 front 2023-07-12T15:46:58.659422+0000 (oldest deadline 2023-07-12T15:47:24.559496+0000) 2023-07-12T15:47:35.851 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:35.514+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.560245+0000 front 2023-07-12T15:47:04.560068+0000 (oldest deadline 2023-07-12T15:47:28.060141+0000) 2023-07-12T15:47:36.131 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:35.864+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:36.132 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:35 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:35.864+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:36.534 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:36 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:36.130+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:36.534 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:36 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:36.130+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:36.534 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:36 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:36.130+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:36.850 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:36 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:36.532+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:58.659592+0000 front 2023-07-12T15:46:58.659422+0000 (oldest deadline 2023-07-12T15:47:24.559496+0000) 2023-07-12T15:47:36.851 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:36 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:36.532+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.560245+0000 front 2023-07-12T15:47:04.560068+0000 (oldest deadline 2023-07-12T15:47:28.060141+0000) 2023-07-12T15:47:36.851 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:36 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5[106183]: 2023-07-12T15:47:36.532+0000 7fa2588df700 -1 osd.5 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:13.361304+0000 front 2023-07-12T15:47:13.361330+0000 (oldest deadline 2023-07-12T15:47:35.661324+0000) 2023-07-12T15:47:37.350 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:37 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:37.113+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:37.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:37 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:37.113+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:37.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:37 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:37.113+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:37.351 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:36 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:36.896+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:37.351 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:36 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:36.896+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:37.352 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:36 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:36.896+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:12.644336+0000 front 2023-07-12T15:47:12.644417+0000 (oldest deadline 2023-07-12T15:47:36.744183+0000) 2023-07-12T15:47:37.956 INFO:journalctl@ceph.osd.5.smithi145.stdout:Jul 12 15:47:37 smithi145 bash[129194]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-5 2023-07-12T15:47:37.956 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:37 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:37.890+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:37.956 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:37 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:37.890+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:37.957 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:37 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:37.890+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:12.644336+0000 front 2023-07-12T15:47:12.644417+0000 (oldest deadline 2023-07-12T15:47:36.744183+0000) 2023-07-12T15:47:38.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:38 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:38.066+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:38.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:38 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:38.066+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:38.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:38 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:38.066+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:39.216 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:38 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:38.867+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:39.216 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:38 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:38.867+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:39.216 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:38 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:38.867+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:12.644336+0000 front 2023-07-12T15:47:12.644417+0000 (oldest deadline 2023-07-12T15:47:36.744183+0000) 2023-07-12T15:47:39.217 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:39 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:39.017+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:39.217 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:39 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:39.017+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:39.217 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:39 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:39.017+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:39.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.5.service' 2023-07-12T15:47:39.263 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:47:39.263 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-12T15:47:39.264 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-12T15:47:39.264 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.6 2023-07-12T15:47:39.600 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:39 smithi145 systemd[1]: Stopping Ceph osd.6 for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:47:39.907 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:39 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:39.646+0000 7f3a0530d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:47:39.907 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:39 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:39.646+0000 7f3a0530d700 -1 osd.6 772 *** Got signal Terminated *** 2023-07-12T15:47:39.907 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:39 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:39.646+0000 7f3a0530d700 -1 osd.6 772 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-12T15:47:40.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:40 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:40.061+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:40.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:40 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:40.061+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:40.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:40 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:40.061+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:40.351 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:39 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:39.905+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:40.351 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:39 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:39.905+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:40.352 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:39 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:39.905+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:12.644336+0000 front 2023-07-12T15:47:12.644417+0000 (oldest deadline 2023-07-12T15:47:36.744183+0000) 2023-07-12T15:47:40.485 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:40 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:40.147Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi069.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2023-07-12T15:47:40.486 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:40 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:40.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi069.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2023-07-12T15:47:40.486 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:40 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:40.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2023-07-12T15:47:41.351 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:40 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:40.935+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:41.351 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:40 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:40.935+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:41.351 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:40 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:40.935+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:12.644336+0000 front 2023-07-12T15:47:12.644417+0000 (oldest deadline 2023-07-12T15:47:36.744183+0000) 2023-07-12T15:47:41.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:41 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:41.111+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:41.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:41 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:41.111+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:41.352 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:41 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:41.111+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:42.351 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:41 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:41.935+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:42.351 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:41 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:41.935+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:42.351 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:41 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:41.935+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:12.644336+0000 front 2023-07-12T15:47:12.644417+0000 (oldest deadline 2023-07-12T15:47:36.744183+0000) 2023-07-12T15:47:42.352 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:42 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:42.098+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:42.352 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:42 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:42.098+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:42.352 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:42 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:42.098+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:43.351 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:42 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:42.944+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:43.351 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:42 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:42.944+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:43.351 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:42 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:42.944+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:12.644336+0000 front 2023-07-12T15:47:12.644417+0000 (oldest deadline 2023-07-12T15:47:36.744183+0000) 2023-07-12T15:47:43.352 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:43.113+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:43.352 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:43.113+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:43.353 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:43.113+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:44.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:44.103+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:44.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:44.103+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:44.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:44 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:44.103+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:44.352 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:43.942+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:57.542045+0000 front 2023-07-12T15:46:57.541994+0000 (oldest deadline 2023-07-12T15:47:19.841769+0000) 2023-07-12T15:47:44.352 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:43.942+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:05.643598+0000 front 2023-07-12T15:47:05.643633+0000 (oldest deadline 2023-07-12T15:47:29.143456+0000) 2023-07-12T15:47:44.352 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:43.942+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:12.644336+0000 front 2023-07-12T15:47:12.644417+0000 (oldest deadline 2023-07-12T15:47:36.744183+0000) 2023-07-12T15:47:44.352 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:43 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6[110720]: 2023-07-12T15:47:43.942+0000 7f39ff8ff700 -1 osd.6 772 heartbeat_check: no reply from 172.21.15.69:6815 osd.3 since back 2023-07-12T15:47:17.245203+0000 front 2023-07-12T15:47:17.245264+0000 (oldest deadline 2023-07-12T15:47:43.145140+0000) 2023-07-12T15:47:45.323 INFO:journalctl@ceph.osd.6.smithi145.stdout:Jul 12 15:47:45 smithi145 bash[129458]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-6 2023-07-12T15:47:45.324 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:45 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:45.070+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:45.324 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:45 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:45.070+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:45.324 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:45 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:45.070+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:46.221 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:46 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:46.102+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:46.221 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:46 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:46.102+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:46.222 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:46 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:46.102+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:46.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.6.service' 2023-07-12T15:47:46.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:47:46.625 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-12T15:47:46.626 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-12T15:47:46.626 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.7 2023-07-12T15:47:46.850 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:46 smithi145 systemd[1]: Stopping Ceph osd.7 for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:47:47.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:47 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:47.002+0000 7f6eb9c0e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:47:47.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:47 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:47.002+0000 7f6eb9c0e700 -1 osd.7 772 *** Got signal Terminated *** 2023-07-12T15:47:47.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:47 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:47.002+0000 7f6eb9c0e700 -1 osd.7 772 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-12T15:47:47.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:47 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:47.142+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:47.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:47 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:47.142+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:47.352 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:47 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:47.142+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:47.352 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:47 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:47.142+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6815 osd.3 since back 2023-07-12T15:47:21.685569+0000 front 2023-07-12T15:47:21.685581+0000 (oldest deadline 2023-07-12T15:47:46.886142+0000) 2023-07-12T15:47:48.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:48 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:48.092+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:48.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:48 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:48.092+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:48.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:48 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:48.092+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:48.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:48 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:48.092+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6815 osd.3 since back 2023-07-12T15:47:21.685569+0000 front 2023-07-12T15:47:21.685581+0000 (oldest deadline 2023-07-12T15:47:46.886142+0000) 2023-07-12T15:47:49.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:49 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:49.067+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:49.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:49 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:49.067+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:49.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:49 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:49.067+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:49.351 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:49 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:49.067+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6815 osd.3 since back 2023-07-12T15:47:21.685569+0000 front 2023-07-12T15:47:21.685581+0000 (oldest deadline 2023-07-12T15:47:46.886142+0000) 2023-07-12T15:47:50.488 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:50 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:50.147Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi069.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2023-07-12T15:47:50.488 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:50 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:50.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi069.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2023-07-12T15:47:50.489 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:47:50 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:47:50.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2023-07-12T15:47:50.601 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:50 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:50.117+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:50.601 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:50 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:50.117+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:50.602 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:50 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:50.117+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:50.602 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:50 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:50.117+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6815 osd.3 since back 2023-07-12T15:47:21.685569+0000 front 2023-07-12T15:47:21.685581+0000 (oldest deadline 2023-07-12T15:47:46.886142+0000) 2023-07-12T15:47:51.600 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:51 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:51.142+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6803 osd.0 since back 2023-07-12T15:46:56.081668+0000 front 2023-07-12T15:46:56.081535+0000 (oldest deadline 2023-07-12T15:47:23.081976+0000) 2023-07-12T15:47:51.601 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:51 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:51.142+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6807 osd.1 since back 2023-07-12T15:47:04.783132+0000 front 2023-07-12T15:47:04.782996+0000 (oldest deadline 2023-07-12T15:47:30.683058+0000) 2023-07-12T15:47:51.601 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:51 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:51.142+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6811 osd.2 since back 2023-07-12T15:47:10.683704+0000 front 2023-07-12T15:47:10.683763+0000 (oldest deadline 2023-07-12T15:47:35.383633+0000) 2023-07-12T15:47:51.601 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:51 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:51.142+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.69:6815 osd.3 since back 2023-07-12T15:47:21.685569+0000 front 2023-07-12T15:47:21.685581+0000 (oldest deadline 2023-07-12T15:47:46.886142+0000) 2023-07-12T15:47:51.601 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:51 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7[115366]: 2023-07-12T15:47:51.142+0000 7f6eb5202700 -1 osd.7 772 heartbeat_check: no reply from 172.21.15.145:6802 osd.4 since back 2023-07-12T15:47:26.886726+0000 front 2023-07-12T15:47:26.886609+0000 (oldest deadline 2023-07-12T15:47:50.386712+0000) 2023-07-12T15:47:52.851 INFO:journalctl@ceph.osd.7.smithi145.stdout:Jul 12 15:47:52 smithi145 bash[129724]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-osd-7 2023-07-12T15:47:53.846 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@osd.7.service' 2023-07-12T15:47:53.887 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:47:53.887 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-12T15:47:53.887 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2023-07-12T15:47:53.888 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@rgw.foo.a 2023-07-12T15:47:54.235 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Jul 12 15:47:53 smithi069 systemd[1]: Stopping Ceph rgw.foo.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:47:54.735 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Jul 12 15:47:54 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-rgw-foo-a[124500]: 2023-07-12T15:47:54.242+0000 7fa69bdae700 -1 received signal: Terminated from /dev/init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-12T15:47:54.736 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Jul 12 15:47:54 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-rgw-foo-a[124500]: 2023-07-12T15:47:54.242+0000 7fa6a0220780 -1 shutting down 2023-07-12T15:48:00.485 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:48:00 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:48:00.148Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi069.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2023-07-12T15:48:00.486 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:48:00 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:48:00.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi069.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2023-07-12T15:48:00.486 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:48:00 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:48:00.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2023-07-12T15:48:04.542 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Jul 12 15:48:04 smithi069 bash[206887]: time="2023-07-12T15:48:04Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2023-07-12T15:48:04.731 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Jul 12 15:48:04 smithi069 bash[206887]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-rgw-foo-a 2023-07-12T15:48:04.731 INFO:journalctl@ceph.rgw.foo.a.smithi069.stdout:Jul 12 15:48:04 smithi069 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2023-07-12T15:48:04.735 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@rgw.foo.a.service' 2023-07-12T15:48:04.772 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:48:04.773 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2023-07-12T15:48:04.773 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2023-07-12T15:48:04.773 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@prometheus.a 2023-07-12T15:48:05.101 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:04 smithi145 systemd[1]: Stopping Ceph prometheus.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:48:05.483 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:48:05.199Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2023-07-12T15:48:05.483 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:48:05.200Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2023-07-12T15:48:05.484 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:48:05.200Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2023-07-12T15:48:05.484 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:48:05.200Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2023-07-12T15:48:05.484 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:48:05.200Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2023-07-12T15:48:05.484 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:48:05.200Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2023-07-12T15:48:05.484 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:48:05.200Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2023-07-12T15:48:05.485 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:48:05.200Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2023-07-12T15:48:05.485 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:48:05.200Z caller=main.go:907 level=info msg="Scrape manager stopped" 2023-07-12T15:48:05.485 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:48:05.201Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2023-07-12T15:48:05.485 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:48:05.201Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2023-07-12T15:48:05.486 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:05 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a[124294]: ts=2023-07-12T15:48:05.201Z caller=main.go:1147 level=info msg="See you next time!" 2023-07-12T15:48:05.486 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Jul 12 15:48:05 smithi145 bash[129995]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-prometheus-a 2023-07-12T15:48:05.667 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@prometheus.a.service' 2023-07-12T15:48:05.705 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-12T15:48:05.706 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2023-07-12T15:48:05.706 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dc75a490-20c7-11ee-9b31-001a4aab830c --force --keep-logs 2023-07-12T15:48:07.236 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:48:07 smithi069 systemd[1]: Stopping Ceph alertmanager.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:48:07.682 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:48:07 smithi069 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a[136560]: ts=2023-07-12T15:48:07.425Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2023-07-12T15:48:07.985 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:48:07 smithi069 bash[207283]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-alertmanager-a 2023-07-12T15:48:08.486 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:48:08 smithi069 systemd[1]: Stopping Ceph node-exporter.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:48:08.487 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:48:08 smithi069 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@alertmanager.a.service: Succeeded. 2023-07-12T15:48:08.487 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Jul 12 15:48:08 smithi069 systemd[1]: Stopped Ceph alertmanager.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:48:08.958 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:48:08 smithi069 bash[207411]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-a 2023-07-12T15:48:09.235 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:48:08 smithi069 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2023-07-12T15:48:09.236 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:48:09 smithi069 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@node-exporter.a.service: Failed with result 'exit-code'. 2023-07-12T15:48:09.236 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Jul 12 15:48:09 smithi069 systemd[1]: Stopped Ceph node-exporter.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:48:09.499 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dc75a490-20c7-11ee-9b31-001a4aab830c --force --keep-logs 2023-07-12T15:48:11.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:48:10 smithi145 systemd[1]: Stopping Ceph iscsi.iscsi.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:48:11.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:48:11 smithi145 ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a[118847]: debug Shutdown received 2023-07-12T15:48:21.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:48:21 smithi145 bash[130333]: time="2023-07-12T15:48:21Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2023-07-12T15:48:21.772 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:48:21 smithi145 bash[130333]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a 2023-07-12T15:48:21.772 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:48:21 smithi145 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2023-07-12T15:48:22.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:48:21 smithi145 bash[130403]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-iscsi-iscsi-a-tcmu 2023-07-12T15:48:22.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:48:21 smithi145 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2023-07-12T15:48:22.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi145.stdout:Jul 12 15:48:21 smithi145 systemd[1]: Stopped Ceph iscsi.iscsi.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:48:22.570 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:48:22 smithi145 systemd[1]: Stopping Ceph grafana.a for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:48:22.852 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:48:22 smithi145 bash[130525]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-grafana-a 2023-07-12T15:48:22.853 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:48:22 smithi145 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@grafana.a.service: Main process exited, code=exited, status=143/n/a 2023-07-12T15:48:23.351 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:48:23 smithi145 systemd[1]: Stopping Ceph node-exporter.b for dc75a490-20c7-11ee-9b31-001a4aab830c... 2023-07-12T15:48:23.351 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:48:22 smithi145 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@grafana.a.service: Failed with result 'exit-code'. 2023-07-12T15:48:23.351 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Jul 12 15:48:22 smithi145 systemd[1]: Stopped Ceph grafana.a for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:48:23.802 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:48:23 smithi145 bash[130649]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c-node-exporter-b 2023-07-12T15:48:23.803 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:48:23 smithi145 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2023-07-12T15:48:24.101 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:48:23 smithi145 systemd[1]: ceph-dc75a490-20c7-11ee-9b31-001a4aab830c@node-exporter.b.service: Failed with result 'exit-code'. 2023-07-12T15:48:24.101 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Jul 12 15:48:23 smithi145 systemd[1]: Stopped Ceph node-exporter.b for dc75a490-20c7-11ee-9b31-001a4aab830c. 2023-07-12T15:48:24.155 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-12T15:48:24.194 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-12T15:48:24.223 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-12T15:48:24.225 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/crash to /home/teuthworker/archive/gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335307/remote/smithi069/crash 2023-07-12T15:48:24.226 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/crash -- . 2023-07-12T15:48:24.265 INFO:teuthology.orchestra.run.smithi069.stderr:tar: /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-12T15:48:24.265 INFO:teuthology.orchestra.run.smithi069.stderr:tar: Error is not recoverable: exiting now 2023-07-12T15:48:24.267 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/crash to /home/teuthworker/archive/gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335307/remote/smithi145/crash 2023-07-12T15:48:24.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/crash -- . 2023-07-12T15:48:24.293 INFO:teuthology.orchestra.run.smithi145.stderr:tar: /var/lib/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-12T15:48:24.294 INFO:teuthology.orchestra.run.smithi145.stderr:tar: Error is not recoverable: exiting now 2023-07-12T15:48:24.294 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-12T15:48:24.295 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-07-12T15:48:24.337 INFO:teuthology.orchestra.run.smithi069.stderr:grep: /var/log/ceph/dc75a490-20c7-11ee-9b31-001a4aab830c/ceph.log: No such file or directory 2023-07-12T15:48:24.339 INFO:tasks.cephadm:Compressing logs... 2023-07-12T15:48:24.339 DEBUG:teuthology.orchestra.run.smithi069:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-12T15:48:24.381 DEBUG:teuthology.orchestra.run.smithi145:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-12T15:48:24.407 INFO:teuthology.orchestra.run.smithi069.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-12T15:48:24.408 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-12T15:48:52.336 INFO:tasks.cephadm:Archiving logs... 2023-07-12T15:48:52.337 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335307/remote/smithi069/log 2023-07-12T15:48:52.338 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-12T15:48:55.060 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/log/ceph to /home/teuthworker/archive/gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335307/remote/smithi145/log 2023-07-12T15:48:55.061 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-12T15:48:57.396 INFO:tasks.cephadm:Removing cluster... 2023-07-12T15:48:57.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dc75a490-20c7-11ee-9b31-001a4aab830c --force 2023-07-12T15:48:58.182 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dc75a490-20c7-11ee-9b31-001a4aab830c --force 2023-07-12T15:48:58.968 INFO:tasks.cephadm:Removing cephadm ... 2023-07-12T15:48:58.969 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-12T15:48:58.989 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-12T15:48:59.006 INFO:tasks.cephadm:Teardown complete 2023-07-12T15:48:59.007 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-12T15:48:59.016 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-12T15:48:59.016 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-12T15:48:59.031 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-12T15:48:59.132 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-12T15:48:59.248 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-12T15:48:59.248 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-12T15:48:59.255 DEBUG:teuthology.orchestra.run.smithi145:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-12T15:48:59.272 INFO:teuthology.orchestra.run.smithi069.stderr:bash: ntpq: command not found 2023-07-12T15:48:59.273 INFO:teuthology.orchestra.run.smithi145.stderr:bash: ntpq: command not found 2023-07-12T15:48:59.277 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-12T15:48:59.277 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2023-07-12T15:48:59.278 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 35 +1022us[+1048us] +/- 90ms 2023-07-12T15:48:59.278 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -801us[ -776us] +/- 68ms 2023-07-12T15:48:59.278 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +6412ns[ +32us] +/- 61ms 2023-07-12T15:48:59.278 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-07-12T15:48:59.279 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-12T15:48:59.279 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2023-07-12T15:48:59.280 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +1041us[+1041us] +/- 90ms 2023-07-12T15:48:59.280 INFO:teuthology.orchestra.run.smithi145.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 -786us[ -778us] +/- 68ms 2023-07-12T15:48:59.280 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +7619ns[+7619ns] +/- 61ms 2023-07-12T15:48:59.280 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-07-12T15:48:59.281 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-12T15:48:59.292 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-12T15:48:59.293 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-12T15:48:59.332 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-12T15:48:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-12T15:48:59.352 INFO:teuthology.orchestra.run.smithi069.stderr:mkdir: cannot create directory ‘/home/ubuntu/cephtest/archive/audit’: No such file or directory 2023-07-12T15:48:59.353 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 1: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' 2023-07-12T15:48:59.353 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-12T15:48:59.372 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-12T15:48:59.415 INFO:teuthology.task.internal:Duration was 2561.041408 seconds 2023-07-12T15:48:59.416 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-12T15:48:59.467 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-12T15:48:59.468 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-12T15:48:59.471 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-12T15:48:59.519 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-12T15:48:59.522 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-12T15:48:59.996 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-12T15:48:59.997 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2023-07-12T15:48:59.998 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-12T15:49:00.021 INFO:teuthology.orchestra.run.smithi069.stderr:grep: /home/ubuntu/cephtest/archive/syslog/kern.log: No such file or directory 2023-07-12T15:49:00.024 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi145.front.sepia.ceph.com 2023-07-12T15:49:00.025 DEBUG:teuthology.orchestra.run.smithi145:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-12T15:49:00.044 INFO:teuthology.orchestra.run.smithi145.stderr:grep: /home/ubuntu/cephtest/archive/syslog/kern.log: No such file or directory 2023-07-12T15:49:00.047 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-12T15:49:00.047 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-12T15:49:00.067 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-12T15:49:00.083 INFO:teuthology.orchestra.run.smithi069.stderr:find: ‘/home/ubuntu/cephtest/archive/syslog’: No such file or directory 2023-07-12T15:49:00.105 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/home/ubuntu/cephtest/archive/syslog’: No such file or directory 2023-07-12T15:49:00.116 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-12T15:49:00.116 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-12T15:49:00.138 DEBUG:teuthology.orchestra.run.smithi145:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-12T15:49:00.153 INFO:teuthology.orchestra.run.smithi069.stderr:bash: /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz: No such file or directory 2023-07-12T15:49:00.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-12T15:49:00.169 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 1: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/internal/syslog.py", line 185, in syslog run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 1: 'sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz' 2023-07-12T15:49:00.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-12T15:49:00.181 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-12T15:49:00.181 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-12T15:49:00.182 INFO:teuthology.orchestra.run.smithi145.stderr:bash: /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz: No such file or directory 2023-07-12T15:49:00.238 INFO:teuthology.orchestra.run.smithi069.stderr:mv: cannot stat '/etc/sudoers.orig.teuthology': No such file or directory 2023-07-12T15:49:00.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-12T15:49:00.239 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 1: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 1: 'sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers' 2023-07-12T15:49:00.240 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-12T15:49:00.254 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-12T15:49:00.282 DEBUG:teuthology.orchestra.run.smithi145:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-12T15:49:00.308 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2023-07-12T15:49:00.309 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = core 2023-07-12T15:49:00.320 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/home/ubuntu/cephtest/archive/coredump’: No such file or directory 2023-07-12T15:49:00.321 INFO:teuthology.orchestra.run.smithi069.stderr:find: ‘/home/ubuntu/cephtest/archive/coredump’: No such file or directory 2023-07-12T15:49:00.322 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/archive/coredump': No such file or directory 2023-07-12T15:49:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-12T15:49:00.323 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/archive/coredump': No such file or directory 2023-07-12T15:49:00.323 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 1: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/internal/__init__.py", line 491, in coredump run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 1: "sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump" 2023-07-12T15:49:00.324 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-12T15:49:00.334 INFO:teuthology.task.internal:Transferring archived files... 2023-07-12T15:49:00.335 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335307/remote/smithi069 2023-07-12T15:49:00.335 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-12T15:49:00.391 INFO:teuthology.orchestra.run.smithi069.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-07-12T15:49:00.391 INFO:teuthology.orchestra.run.smithi069.stderr:tar: Error is not recoverable: exiting now 2023-07-12T15:49:00.392 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 1: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.8/tarfile.py", line 2313, in next tarinfo = self.tarinfo.fromtarfile(self) File "/usr/lib/python3.8/tarfile.py", line 1103, in fromtarfile obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors) File "/usr/lib/python3.8/tarfile.py", line 1039, in frombuf raise EmptyHeaderError("empty header") tarfile.EmptyHeaderError: empty header During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/misc.py", line 749, in pull_directory tar = tarfile.open(mode='r|', fileobj=r.stdout) File "/usr/lib/python3.8/tarfile.py", line 1629, in open t = cls(name, filemode, stream, **kwargs) File "/usr/lib/python3.8/tarfile.py", line 1510, in __init__ self.firstmember = self.next() File "/usr/lib/python3.8/tarfile.py", line 2328, in next raise ReadError("empty file") tarfile.ReadError: empty file 2023-07-12T15:49:00.395 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-12T15:49:00.405 INFO:teuthology.task.internal:Not uploading archives. 2023-07-12T15:49:00.405 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-12T15:49:00.434 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-12T15:49:00.435 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-12T15:49:00.438 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-12T15:49:00.453 INFO:teuthology.orchestra.run.smithi069.stdout: 262320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 12 15:48 /home/ubuntu/cephtest 2023-07-12T15:49:00.454 INFO:teuthology.orchestra.run.smithi145.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 12 15:48 /home/ubuntu/cephtest 2023-07-12T15:49:00.456 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-12T15:49:00.466 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-12T15:49:00.503 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi069 with status 1: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' 2023-07-12T15:49:00.504 INFO:teuthology.nuke:Checking targets against current locks 2023-07-12T15:49:00.534 DEBUG:teuthology.nuke:shortname: smithi069 2023-07-12T15:49:00.535 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-12T15:49:00.558 DEBUG:teuthology.nuke:shortname: smithi145 2023-07-12T15:49:00.558 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-12T15:49:00.562 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335307', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 15:00:50.299139', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-12T15:49:00.583 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2023-07-12_13:54:22-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335307', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 15:00:50.300999', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-12T15:49:00.614 INFO:teuthology.orchestra.console:Power off smithi069 2023-07-12T15:49:00.615 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-12T15:49:00.638 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T15:49:00.662 INFO:teuthology.orchestra.console:Power off smithi145 2023-07-12T15:49:00.663 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-12T15:49:00.689 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T15:49:04.677 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T15:49:04.718 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T15:49:08.792 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T15:49:08.830 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T15:49:09.004 INFO:teuthology.orchestra.console:Power off for smithi069 completed 2023-07-12T15:49:09.042 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2023-07-12T15:49:09.148 INFO:teuthology.run:Summary data: description: orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} duration: 2561.041407585144 failure_reason: 'Command failed on smithi069 with status 1: ''mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log''' flavor: default owner: scheduled_gabrioux@teuthology status: fail success: false 2023-07-12T15:49:09.148 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-12T15:49:09.243 INFO:teuthology.run:FAIL